Task #13271

Investigate options to reduce memory usage working with related models

Added by José Raddaoui Marín over 1 year ago.

Status:NewStart date:03/09/2020
Priority:MediumDue date:
Assignee:-% Done:

0%

Category:Data model / ORM
Target version:-
Google Code Legacy ID: Tested version:1.2, 1.3, 1.3.1, 2.0.0, 2.0.1, 2.1, 2.1.1, 2.1.2, 2.2, 2.2.1, 2.3, 2.4, 2.5, 2.6, pre-1.2
Sponsored:No Requires documentation:

Description

In the latest scalability enhancements we are adding calls to Qubit::clearClassCaches(); in places where the related resources ($this->informationObjects for example) are looped. This is added to avoid high memory usage while the related resources are hydrated. E.g.:

https://github.com/artefactual/atom/commit/5981406ad5e0b327c7c8ac957b3c9290608943ed

I wonder what are the side effects of calling this function several times. Does it cause a performance hit? Do we want to keep other classes in cache? It's possible that the answers to those questions don't justify the changes required to implement this is a different manner (specially the one outlined below), but we should investigate if it's possible to do this better/cleaner.

Laravel 6 introduced lazy collections (source code), using PHP's generators to work with large datasets while keeping the memory usage low. Implementing this in the good old Symfony and Propel 1 may not be an easy task, but the initial ideal would be to include it in the Qubit model classes (through the models builder). It doesn't have to replace the current implementation entirely, it could add an extra method/property (e.g.: $this->informationObjectsCursor) to be used in loops only, while keeping $this->informationObjects for other uses where the generator may be an issue.


Related issues

Related to Access to Memory (AtoM) - Task #13355: Reduce memory usage on descriptions CSV import Verified 06/15/2020

History

#1 Updated by José Raddaoui Marín over 1 year ago

  • Related to Task #13355: Reduce memory usage on descriptions CSV import added

Also available in: Atom PDF