Skip to content

Commit

Permalink
Cut down on unnecssary HTTP requests
Browse files Browse the repository at this point in the history
Set `Search.num_found` on first search request made, so that if
it is called after this an unnecessary request to get the total
number of docs found by search is avoided.
  • Loading branch information
jake authored and jake committed May 27, 2022
1 parent 7b60d72 commit c7ab48f
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions internetarchive/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,9 @@ def _advanced_search(self):
auth=self.auth,
**self.request_kwargs)
j = r.json()
num_found = int(j['response']['numFound'])
if not self._num_found:
self._num_found = num_found
if j.get('error'):
yield j
yield from j.get('response', {}).get('docs', [])
Expand All @@ -148,6 +151,8 @@ def _scrape(self):
if j.get('error'):
yield j
num_found = int(j['total'])
if not self._num_found:
self._num_found = num_found
self._handle_scrape_error(j)

self.params['cursor'] = j.get('cursor')
Expand Down

0 comments on commit c7ab48f

Please sign in to comment.