Describe the bug
A clear and concise description of what the bug is.

Steampipe version (steampipe -v)
Example: v0.3.0

Plugin version (steampipe plugin list)
Example: v0.5.0

To reproduce
Steps to reproduce the behavior (please include relevant code and/or commands).

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Joining these tables on key columns always returns no records:

• github_repository_issue
• github_user

Joining these tables on key columns doesnt as expected:

• github_gist
• github_organization

This works:

select
  repository_full_name,
  issue_number,
  title
from
  github_repository_issue
where 
  not is_pull_request 
  and  repository_full_name = 'turbot/steampipe-plugin-github'

This should work, but does not:

select
  i.repository_full_name,
  i.issue_number,
  i.title
from
  github_repository as r,
  github_repository_issue as i
where 
  not i.is_pull_request 
  and  i.repository_full_name = r.full_name

This is due to SDK bug turbot/steampipe-postgres-fdw#3

References

List daily traffic view statistics for a repository.

similar to #71

Is your feature request related to a problem? Please describe.

The collaborators columns on github_repository currently include all collaborators - both team members (internal) and outside collaborators. This is helpful, but there is no way to tell them apart and check for outside collaborators only.

Describe the solution you'd like

Add a new outside_collaborators column, which is just the outside collaborators for the repo.

Describe alternatives you've considered

Tried to filter the collaborators, but the API does not return any way to tell them apart.

Describe the bug
When selecting from github_branch, passing a repository_full_name with no slash in results in index out of range panic

 select * from github.github_branch where repository_full_name = 'steampipe'
Error: rpc error: code = Internal desc = list call tableGitHubBranchList failed with panic runtime error: index out of range [1] with length 1

Steampipe version (steampipe -v)
v0.8.0

Plugin version (steampipe plugin list)
v0.6.0

To reproduce

select * from github.github_branch where repository_full_name = 'steampipe'

Expected behavior
An empty result set

Additional context
This does not give the crash

select * from github.github_branch where repository_full_name = 'steampipe/'

References

List all branches in a repository.

Currenting throwing an error on github_user table as follows:

> select * from github_user
Error: pq: rpc error: code = Internal desc = 'List' call requires an '=' qual for column: login

Same when using a SQL client with the service. Confirmed the Access Token has the required scopes.

Also small docs note: https://hub.steampipe.io/plugins/turbot/github

scope for read:email should be user:email I think?

Describe the bug
This information is already captured in the members_can_create_repos column and all of the data for members_can_create_repositories comes through as null today anyway.

Steampipe version (steampipe -v)
v0.5.0

Plugin version (steampipe plugin list)
v0.4.0

To reproduce
select * from github_my_organization;

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

It's hard to make sense of the table_github_gist listing without filenames. I found here the element Files.

I tried adding this to the table_github_gist.go:

{Name: "files", Type: pb.ColumnType_JSON, Description: "The filename."},

Now I can do this:

> select
  files->jsonb_object_keys(files)->'filename' as name,
  files->jsonb_object_keys(files)->'language' as language,
  description
from github_my_gist limit 5
+------------------------------+--------------+----------------------------------------------------+
| name                         | language     | description                                        |
+------------------------------+--------------+----------------------------------------------------+
| "ucdavis-april-2019.md"      | "Markdown"   |                                                    |
| "plpython.md"                | "Markdown"   |                                                    |
| "internal_and_public_id"     | <null>       | convert between hypothesis internal and public ids |
| "addMembers.js"              | "JavaScript" |                                                    |
| "async-postgres-listener.py" | "Python"     |                                                    |
+------------------------------+--------------+----------------------------------------------------+

The downside is that this requires some Postgres mojo that makes me stop and think, despite my having used Postgres JSONB a lot. So while this works for me I'm not sure I'd recommend for others. Presumably the plugin could use a derived type that hoists these fields to the top level? Anyway, it was an easy and instructive workaround.

Describe the bug

Docs for github_branch have a copy and paste error in the description.

The login I should've used was boto but I guessed amazon and got this result:

select * from github_organization where login = 'amazon';

ERROR: rpc error: code = Internal desc = hydrate function getOrganizationDetail failed with panic runtime error: invalid memory address or nil pointer dereference

References

List all workflows for a given repository.

I tried the first example at https://hub.steampipe.io/plugins/turbot/github/tables/github_issue.

select
  sha,
  author_login,
  author_date,
  message
from
  github_commit
where
  repository_full_name = 'turbot/steampipe'
order by
  author_date desc;

Result in 0.8.2:

ERROR: invalid input syntax for type json
DETAIL: Escape sequence "'" is invalid.
CONTEXT: JSON data, line 1: ...pe-postgre-fdw to v0.2.2 - fix JSON data with ''...

Does this repro for others?

This works.

select
  *
from
  github_commit
where
  repository_full_name = 'turbot/steampipe' 
limit 1;

I was using this code to load 88,545 commits into a table that I still have in Steampipe.

create table commits as (
  select
    *
  from
    github_commit g
  where
    g.repository_full_name = 'microsoft/vscode'
);

So this was clearly possible within the 5K/hr API rate limit, and I did it multiple times over several days.

In the course of that, I triggered a reset a couple of times.

Now, the above code only gets ~2200 commits, and costs me about that many calls to get them, per https://api.github.com/rate_limit. I'll never get 88K commits again at that rate!

    "rate": {
        "limit": 5000,
        "used": 2270,
        "remaining": 2730,
        "reset": 1634539423
    }

What could have changed? I don't see how it's Steampipe but maybe it's us? Could GitHub be punishing me for infractions? Could they have imposed some other kind of general limit between Friday and Sunday? Should someone with a different GitHub personal access token try to repro this? And finally, @nathan, should this be a separate bug report vs an addendum to this feature request?

I just checked this again. Got only two steps into a dozen-stage pipeline that was running successfully, and repeatedly, until last night.

create table if not exists vscode_log(time timestamp, event text);

insert into vscode_log(time, event) values (now(), 'vscode_org_members');
-- ~50 api calls
create table vscode_org_members as (
  select
    g.name,
    g.login,
    jsonb_array_elements_text(g.member_logins) as member_login
  from
    github_organization g
  where
    g.login = 'microsoft'
);

insert into vscode_log(time, event) values (now(), 'vscode_commits');
-- ~1000 API calls for 1000 records: 1 call per commit?!?
create table vscode_commits as (
  select
    *
  from
    github_commit g
  where
    g.repository_full_name = 'microsoft/vscode'
  limit 1000
);

Postman:

"rate": {
        "limit": 5000,
        "used": 1108,
        "remaining": 3892,
        "reset": 1634578299
    }

I'm stuck. I built and tested a pipeline, have all the data from it sitting in local tables, wrote an article about all that, intending to publish the pipeline and encourage folks to try it for themselves, but now I cannot myself run the pipeline.

Update: I can get 100 commits per API call going straight to
https://api.github.com/repos/microsoft/vscode/commits?per_page=100.

Tried rolling back to plugin .6 with older sdk, still the same.

select
  full_name,
  private,
  visibility
from
  github.github_repository;

gives error

 'List' call requires an '=' qual for column: full_name

References
Add any related links that will help us understand the resource, including vendor documentation, related GitHub issues, and Go SDK documentation.

v0.8.0

Succeed (24 rows): select jsonb_array_elements(members) from github_organization where login = 'turbot'

Fail: (0 rows): select jsonb_array_elements(member_logins) from github_organization where login = 'turbot'

v0.7.0: both yield 24 rows

References

I'd like a table to list all the releases and their assets for a respository.

References

List the details of the various gitignore templates GitHub defines.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

I'm having to check mine manually using Postman, would be nice to have this in Steampipe.

Describe the bug
On running the query

select * from github_commit where repository_full_name='turbot/steampipe'

Output:

Error: invalid input syntax for type json

Steampipe version (steampipe -v)
Example: v0.8.1

Plugin version (steampipe plugin list)
Example: v0.6.0

To reproduce
Steps to reproduce the behaviour (please include relevant code and/or commands).

Expected behaviour
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

References

List all tags in a repository.

It would be great if the GitHub plugin supported GitHub Enterprise. This would be mostly a change to the config and connection setup to allow a custom host etc.

After development, we'll require someone with access to GitHub Enterprise to help with final testing.

References

Return community profile information for a repository per https://docs.github.com/en/rest/reference/repos#get-community-profile-metrics

References

List all stargazers of a repository.

References

Return all commits for a given repository.

References

I'd like a table to get the branch protection rules for a repository.

Is your feature request related to a problem? Please describe.
When retrying API calls due to errors, today we use custom retry logic that sometimes fails or runs inefficiently.

Describe the solution you'd like
We should switch to using the RetryHydrate function from the plugin SDK for better error handling/retry logic, e.g., f7142a2.

Describe alternatives you've considered
N/A

Additional context
N/A

References

List weekly traffic view statistics for a repository.

GitHub GraphQL API supports DependencyGraph* resources for quite a few package managers now.

https://docs.github.com/en/graphql/reference/objects#dependencygraphdependency

Doesn't look like go-github will be supporting it (as that relies on the REST API).