Bug #12676

Improve CSV Import note creation logic performance

Added by Steve Breker over 3 years ago. Updated over 3 years ago.

Status:VerifiedStart date:12/27/2018
Priority:MediumDue date:
Assignee:Steve Breker% Done:


Category:CSV import
Target version:Release 2.5.0
Google Code Legacy ID: Tested version:
Sponsored:No Requires documentation:


There is code in the csv:import logic that prevents duplicate notes from being created if:
- they already exist in the database for this object, culture and note type.
- the same note content, and type is duplicated in the import file.

This logic was working, but is extremely slow since it goes to the database to retrieve all existing notes for every note in each CSV line that is being added. This could mean 100s or 1000s of extra requests in cases where there are a lot of notes.

Change the logic so that it retrieves all existing notes once only, then keeps track of the ones that are currently being added to ensure no dupes are created.

Resulting functionality should be exactly the same as before:
- new notes being imported should not be duplicated if the note exists already
- if the same note is included in an import file more than once, do not create duplicate notes.


#2 Updated by Mike Cantelon over 3 years ago

  • Status changed from In progress to Feedback

#3 Updated by Dan Gillean over 3 years ago

  • Status changed from Feedback to Verified

Also available in: Atom PDF