Task #13384

Reduce memory usage on nested set build task

Added by José Raddaoui Marín about 1 year ago. Updated 6 months ago.

Status:VerifiedStart date:07/16/2020
Priority:MediumDue date:
Assignee:Miguel Angel Medinilla Luque% Done:

0%

Category:Performance / scalability
Target version:Release 2.6.3
Google Code Legacy ID: Tested version:
Sponsored:No Requires documentation:

Description

The nested set build task saves all statements on memory leading to a high memory usage in databases with a considerable amount of resources (500K IOs -> ~500 MB). Moreover, the task is executed automatically on CSV imports after #13355, see Miguel Angel's analysis from that ticket:

https://projects.artefactual.com/issues/13355#note-5


Related issues

Related to Access to Memory (AtoM) - Task #13355: Reduce memory usage on descriptions CSV import Verified 06/15/2020

History

#1 Updated by José Raddaoui Marín about 1 year ago

  • Related to Task #13355: Reduce memory usage on descriptions CSV import added

#2 Updated by José Raddaoui Marín 8 months ago

  • Status changed from New to Code Review
  • Assignee set to Steve Breker

Ready for code review in https://github.com/artefactual/atom/pull/1241.

I tested these changes with a considerable big database (915875 IOs, 347624 terms and 64 menus):

- Max memory usage was reduced from 670 to 175 MB.
- Execution time was increased from 100 to 132 seconds.
- CPU usage was a bit more even between MySQL and the PHP task.

#3 Updated by Steve Breker 8 months ago

  • Status changed from Code Review to Feedback
  • Assignee changed from Steve Breker to José Raddaoui Marín

CR complete - looks great Radda!

Ran locally with no issues.

#4 Updated by José Raddaoui Marín 8 months ago

  • Status changed from Feedback to QA/Review
  • Assignee changed from José Raddaoui Marín to Miguel Angel Medinilla Luque

Thanks Steve!

Merged in qa/2.x (7fedc9c). Miguel Angel. could you repeat the test from #13355 to verify this changes? Testing a CSV import over a considerable big database would be great, but if you can only test the nested set task, I think that would be okay.

#5 Updated by Redmine Admin 8 months ago

  • Target version changed from Release 2.6.2 to Release 2.6.3

#6 Updated by Sarah Mason 6 months ago

Verified with rebuild-nested-set task. Spoke to Miguel about testing over a large database.
-Sarah M.

#7 Updated by Sarah Mason 6 months ago

  • Status changed from QA/Review to Verified

Also available in: Atom PDF