Avoid pagination over 10,000 records in ES queries - search/browse result pages over 1,000 lead to ES error
|Category:||Search / Browse|
|Target version:||Release 2.5.0|
|Google Code Legacy ID:||Tested version:||2.5|
Elasticsearch introduced a new setting in 2.x to avoid memory issues in deep pagination:
index.max_result_window The maximum value of from + size for searches to this index. Defaults to 10000. Search requests take heap memory and time proportional to from + size and this limits that memory. See Scroll or Search After for a more efficient alternative to raising this.
This means that for uses with 10,000+ records in an AtoM installation, trying to navigate to any page above 1,000 in the search/browse results (when the results per page setting is set to the default value of 10) will lead to an Elasticsearch error.
Using the Scroll or Search After APIs are not options for us because they don't allow to go to an intermediate page and the Scroll API doesn't return aggregations.
Increasing the index.max_result_window value also increases the cost of the queries and we have instances with over a million descriptions.
Therefore, we'll limit the pagination in ES queries to 10000 records, showing a note about the issue in the pages over that limit. To ensure that users can still access those results if needed, we will also add the ability to specify the sort order in the sort button - for more details see subtask #12000.
See also the following notes from the Elastic team about this issue:
#6 Updated by José Raddaoui Marín about 1 year ago
Places using ES pagination and affected by this issue:
- IO browse page
- Actor browse page
- Repository browse page
- Accessions browse page
- Move action
- Description updates
- Actor index page (related descriptions list)
- Repository index page (holdings)
- Repository index page (maintained actors)
- IO inventory report
- Taxonomy index page
- Term index page (IO results)
- Term index page (terms list)
- API: IO browse endpoint
#12 Updated by Dan Gillean about 1 year ago
- File pager-results-redirect.png added
- File pager-results-redirect-02.png added
- Status changed from QA/Review to Verified
- Requires documentation set to Yes
Looks good! Attaching a couple images of the notification shown when redirected back to the first page of results, for reference and for use in the docs.