Bug #13463

Problem: CSV import with pipe separated culture data crashes import and adds bad data

Added by David Juhasz 9 months ago. Updated 9 months ago.

Status:NewStart date:01/21/2021
Priority:MediumDue date:
Assignee:-% Done:

0%

Category:CSV import
Target version:-
Google Code Legacy ID: Tested version:2.6
Sponsored:No Requires documentation:

Description

Attempting to import archival description data with a CSV import file that has pipe separated values in the "culture" column causes the import to halt immediately with an error, and adds bad data to the AtoM databases.

To reproduce

1) Log in to AtoM as an administrator
2) Select "Import > CSV"
3) With the default options, import the attached "atom_import_test_rad.csv" file
4) Click the job link to view the job output

Errors encountered

PHP Fatal Error:

PHP Notice:  Undefined index: en|fr in
/usr/share/nginx/atom/lib/task/import/csvImportTask.class.php on line 1009; 
PHP Warning:  array_map(): Expected parameter 2 to be an array, null given in
/usr/share/nginx/atom/lib/task/import/csvImportBaseTask.class.php on line 563;
PHP Warning:  array_search() expects parameter 2 to be array, null given in
/usr/share/nginx/atom/lib/task/import/csvImportBaseTask.class.php on line 563;
PHP Notice:  Undefined index: en|fr in
/usr/share/nginx/atom/lib/task/import/csvImportTask.class.php on line 563; 
PHP Warning:  array_map(): Expected parameter 2 to be an array, null given in
/usr/share/nginx/atom/lib/task/import/csvImportBaseTask.class.php on line 563;
PHP Warning:  array_search() expects parameter 2 to be array, null given in
/usr/share/nginx/atom/lib/task/import/csvImportBaseTask.class.php on line 563;
Publication status: 'published' is invalid. Using default.;   
Unable to execute INSERT statement. [wrapped: SQLSTATE[23000]: Integrity 
constraint violation: 1452 Cannot add or update a child row: a foreign key 
constraint fails (atom.object_term_relation, CONSTRAINT 
object_term_relation_FK_3 FOREIGN KEY (term_id) REFERENCES term (id) ON DELETE 
CASCADE)]

Bad MySQL data:

mysql> select io.id, io.source_culture, i18n.culture from information_object io
join information_object_i18n i18n on io.id = i18n.id where io.id=2074053;
+---------+----------------+-----------+
| id      | source_culture | culture   |
+---------+----------------+-----------+
| 2074053 | en|fr          | * en
* fr |
| 2074053 | en|fr          | en|fr     |
+---------+----------------+-----------+
2 rows in set (0.00 sec)

Expected results

  • Each CSV rows with an invalid culture value is skipped, and the row data not imported into AtoM
  • The user is notified that the row has been skipped because of an invalid culture value

Determining what culture values are valid in the import context could be controversial, depending on the criteria chosen. While non-alphabetic characters like the pipe (|) should be non-controversial, ISO codes could be limited to the languages enabled in the "i18n language" admin setting, all ISO 639-1 language codes, or all ISO 639-3 language codes. The "I18n language" setting language list includes ISO 639-1 two letter codes and ISO 639-3 three letter codes, but not all languages can be selected for some reason.

atom_import_test_rad.csv Magnifier - RAD CSV import file with bad "culture" column data (4.16 KB) David Juhasz, 01/21/2021 08:20 PM


Related issues

Related to Access to Memory (AtoM) - Bug #13181: Languages with 3-letter codes cannot be added in AtoM's u... In progress 09/24/2019

History

#1 Updated by David Juhasz 9 months ago

  • Subject changed from Problem: CSV import with pipe separated culture value crashes import to Problem: CSV import with pipe separated culture value crashes import and adds bad data

#2 Updated by David Juhasz 9 months ago

  • Subject changed from Problem: CSV import with pipe separated culture value crashes import and adds bad data to Problem: CSV import with pipe separated culture data crashes import and adds bad data

#3 Updated by Dan Gillean 9 months ago

David: FYI, if you're curious about the issue with the languages listed in Symfony and why some can't be selected, Radda investigated this a while ago and added some explanations on #13181.

#4 Updated by David Juhasz 9 months ago

  • Related to Bug #13181: Languages with 3-letter codes cannot be added in AtoM's user interface settings added

Also available in: Atom PDF