Task #13525

Optimize physicalobject:normalize task

Added by José Raddaoui Marín about 1 month ago.

Status:NewStart date:06/16/2021
Priority:LowDue date:
Assignee:-% Done:

0%

Category:CLI tools
Target version:-
Google Code Legacy ID: Tested version:2.6
Sponsored:No Requires documentation:

Description

When running the physicalobject:normalize task over a considerable big amount of physical objects (360K), the task reports the counts and it gets killed after a couple of hours of heavy CPU use:

root@atom:/usr/share/nginx/atom# php -d memory_limit=-1 symfony physicalobject:normalize

 Are you sure you'd like to normalize physical object data?
y
Data before clean-up:
 - 360781 physical objects
 - 360780 physical object relations

Detecting duplicates and updating relations to duplicates...
(Using physical object name, location, and type as the basis for duplicate detection.)
root@atom:/usr/share/nginx/atom# php -d memory_limit=-1 symfony physicalobject:normalize

 Are you sure you'd like to normalize physical object data?
y
Data before clean-up:
 - 360781 physical objects
 - 360780 physical object relations

Detecting duplicates and updating relations to duplicates...
(Using physical object name, location, and type as the basis for duplicate detection.)
Killed

From ... https://groups.google.com/g/ica-atom-users/c/5TKE4sJN-Q8/m/eYJTymlDAgAJ

I added some extra logging and looks like the task is running properly and, while I couldn't get to the Killed point, it took a lot of time to process just a part of the physical objects. This could be improved by using raw SQL for the deletions and relation updates, and changing the logic a bit to reduce the amount of queries.

Also available in: Atom PDF