Rate limit error in 1.0.143

[ksrisurapaneni]

Describe the bug
We were using 1.0.132 before, starting using 1.0.143 since Friday. Our process iterates across 5 Git organizations and creates PRs in each if it finds a match. It was running fine in 1.0.132, but fails in 1.0.143 after iterating through 1 or 2 Git orgs with the rate limit exception.

Exception in thread "main" org.kohsuke.github.GHException: Failed to retrieve https://api.github.com/search/code?q=filename%3ADockerfile+user%3Acbsi-cmg+FROM+xyz+container+pipeline+docker.jfrog.io%2Frv-python-3+8-buster
	at org.kohsuke.github.GitHubPageIterator.fetch(GitHubPageIterator.java:155)
	at org.kohsuke.github.GitHubPageIterator.hasNext(GitHubPageIterator.java:91)
	at org.kohsuke.github.PagedSearchIterable$1.hasNext(PagedSearchIterable.java:86)
	at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:106)
	at org.kohsuke.github.PagedIterator.hasNext(PagedIterator.java:74)
	at org.kohsuke.github.PagedSearchIterable.populate(PagedSearchIterable.java:65)
	at org.kohsuke.github.PagedSearchIterable.getTotalCount(PagedSearchIterable.java:49)
	at com.salesforce.dockerfileimageupdate.utils.DockerfileGitHubUtil.findFilesWithImage(DockerfileGitHubUtil.java:88)
	at com.salesforce.dockerfileimageupdate.utils.DockerfileGitHubUtil.getGHContents(DockerfileGitHubUtil.java:312)
	at com.salesforce.dockerfileimageupdate.subcommands.impl.Parent.execute(Parent.java:61)
	at com.salesforce.dockerfileimageupdate.CommandLine.main(CommandLine.java:52)
Caused by: org.kohsuke.github.HttpException: {
  "documentation_url": "https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#abuse-rate-limits",
  "message": "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."
}

Can you please look into what changed that we are reaching rate limits so soon?

Thanks,
Keerti

[justinharringa]

Hi @ksrisurapaneni have you had a chance to take a look at https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#abuse-rate-limits (referenced in the error)? Per previous concerns, we're expanding your search now so you may hit many more results.

Would it be possible to include the commands you ran and a rough idea of how many repositories you have in each org?

Hope all is well

[ksrisurapaneni]

Hi Justin,

One of our largest orgs has around ~5k repos. I was wondering if you could add a sleep and retry condition if this error occurs.

Rough code in Python that I am using in a different use case:

        for i in range(0, int(math.ceil(result.totalCount / 30))):
            try:
                files = result.get_page(i)
                # print(files)
            except StopIteration:
                break
            except RateLimitExceededException:
                search_rate_limit = g.get_rate_limit().search
                logger.warning(f'{search_rate_limit.remaining} API calls remaining')
                reset_timestamp = calendar.timegm(search_rate_limit.reset.timetuple())
                # add 10 seconds to be sure the rate limit has been reset
                sleep_time = reset_timestamp - calendar.timegm(time.gmtime()) + 10
                time.sleep(sleep_time)
                files = result.get_page(i)

Thanks,
Keerti