I was using this code to load 88,545 commits into a table that I still have in Steampipe.
create table commits as (
select
*
from
github_commit g
where
g.repository_full_name = 'microsoft/vscode'
);
So this was clearly possible within the 5K/hr API rate limit, and I did it multiple times over several days.
In the course of that, I triggered a reset a couple of times.
Now, the above code only gets ~2200 commits, and costs me about that many calls to get them, per https://api.github.com/rate_limit. I'll never get 88K commits again at that rate!
"rate": {
"limit": 5000,
"used": 2270,
"remaining": 2730,
"reset": 1634539423
}
What could have changed? I don't see how it's Steampipe but maybe it's us? Could GitHub be punishing me for infractions? Could they have imposed some other kind of general limit between Friday and Sunday? Should someone with a different GitHub personal access token try to repro this? And finally, @nathan, should this be a separate bug report vs an addendum to this feature request?
I just checked this again. Got only two steps into a dozen-stage pipeline that was running successfully, and repeatedly, until last night.
create table if not exists vscode_log(time timestamp, event text);
insert into vscode_log(time, event) values (now(), 'vscode_org_members');
-- ~50 api calls
create table vscode_org_members as (
select
g.name,
g.login,
jsonb_array_elements_text(g.member_logins) as member_login
from
github_organization g
where
g.login = 'microsoft'
);
insert into vscode_log(time, event) values (now(), 'vscode_commits');
-- ~1000 API calls for 1000 records: 1 call per commit?!?
create table vscode_commits as (
select
*
from
github_commit g
where
g.repository_full_name = 'microsoft/vscode'
limit 1000
);
Postman:
"rate": {
"limit": 5000,
"used": 1108,
"remaining": 3892,
"reset": 1634578299
}
I'm stuck. I built and tested a pipeline, have all the data from it sitting in local tables, wrote an article about all that, intending to publish the pipeline and encourage folks to try it for themselves, but now I cannot myself run the pipeline.
Update: I can get 100 commits per API call going straight to
https://api.github.com/repos/microsoft/vscode/commits?per_page=100.
Tried rolling back to plugin .6 with older sdk, still the same.