Feature #8572

Allow importing of multiple languages for the same record in csv:import

Added by Mike Gale over 4 years ago. Updated 13 days ago.

Status:FeedbackStart date:06/17/2015
Priority:MediumDue date:
Assignee:Mike Cantelon% Done:

0%

Category:CSV importEstimated time:32.00 hours
Target version:Release 2.6.0
Google Code Legacy ID: Tested version:2.4
Sponsored:Yes Requires documentation:Yes

Description

Currently, if a record has multiple translations and you export it, it will produce a row for each language. However, on import our csv:import task will throw an error "duplicate legacyId, skipping" instead of supplementing the existing record with the translation.

We should add functionality so if a previous row had the same legacyId as the current one, but a different culture, we should supplement the already imported record with the new language data.

Desired final behavior

  • If 2 or more rows in the same CSV have the same legacyID and the same culture values, AtoM will throw an error and halt the import, or throw a warning and skip the second row.
  • If 2 or more rows in the same CSV have the same legacyID values and different culture values, the subsequent rows will be added as a translation of the first row.

example_information_objects_isad_2.3.csv Magnifier - Test CSV, 2018-10-30 (8.74 KB) Dan Gillean, 10/31/2018 07:45 AM

other-languages.png (59.3 KB) Dan Gillean, 08/16/2019 02:14 PM

large-items-collection.csv Magnifier (344 KB) Dan Gillean, 08/16/2019 02:17 PM

rowlf-translation-test.csv Magnifier (4.22 KB) Dan Gillean, 08/23/2019 09:33 AM

History

#1 Updated by Mike Gale over 4 years ago

  • Status changed from New to Invalid
  • Requires documentation set to Yes

Apparently AtoM already supports this feature.

#2 Updated by Dan Gillean 12 months ago

  • File example_information_objects_isad_2.3.csvMagnifier added
  • Project changed from Access to Memory (AtoM) to AtoM Wishlist
  • Category deleted (CSV import)
  • Status changed from Invalid to New
  • Target version deleted (Release 2.3.0)
  • Tested version 2.4 added

Tested in 2.4 vagrant box, and by a user in the forum, 2018-10-30 (forum post here ). This no longer works as described, and may perhaps have been removed during the 2.4 overhaul of the CSV import?

Tested both CLI import and UI. Attaching a CSV used for testing. It contains a parent description and several item-level children. 3 of the child items have duplicate rows with the same legacyID but different culture values (and data only added in those rows to translatable free-text fields). Expected results, based on previous functionality and what is described in this ticket was:

  • If 2 rows in the same CSV have the same legacyID and the same culture values, AtoM will throw an error and halt the import
  • If 2 rows in the same CSV have the same legacyID values and different culture values, the second/lower row will be added as a translation of the first.

What actually happened was that culture was ignored, no errors or warnings were thrown for the duplicate legacyID values, and all rows were imported as new records.

Moving this ticket to the WishList so we can consider adding it back, or similar functionality in the future.

#3 Updated by Dan Gillean 12 months ago

  • Assignee deleted (Mike Gale)

#4 Updated by Mike Cantelon 3 months ago

Sketched this functionality out to make sure it can work, for future reference: dev/experiment/csv-translation-import (UPDATE: deleted and moved to dev/issue-8572-csv-import-translation).

#6 Updated by Dan Gillean 3 months ago

  • Estimated time set to 32.00

#7 Updated by Dan Gillean 3 months ago

  • Project changed from AtoM Wishlist to Access to Memory (AtoM)
  • Description updated (diff)
  • Category set to CSV import
  • Assignee set to Mike Cantelon
  • Target version set to Release 2.6.0
  • Sponsored changed from No to Yes
  • Requires documentation set to Yes

#9 Updated by Mike Cantelon 2 months ago

  • Status changed from New to Code Review
  • Assignee deleted (Mike Cantelon)

#10 Updated by Steve Breker 2 months ago

  • Status changed from Code Review to Feedback
  • Assignee set to Mike Cantelon

CR complete. Looks good!

#11 Updated by Mike Cantelon 2 months ago

  • Status changed from Feedback to QA/Review
  • Assignee deleted (Mike Cantelon)

Thanks Steve!

Merged into qa/2.6.x.

#12 Updated by Dan Gillean 2 months ago

Hi Mike!

A couple of things I wanted your feedback on.

First, importing the sample currently attached to this ticket from a couple years ago:

  • I noticed that immediately following import via the UI (which should index progressively), that the "Other language(s) available" button wasn't showing on descriptions imported with a translation.
  • I then re-indexed, cleared the cache, and restarted services - which did make them show up, but there was weird behavior: other records were being shown in the drop-down, and when clicked, it would flip the UI but also take the user to the other record. It should be showing available translations for the current record. See the attached screenshot for an example.

Second, I tried adding a few translations rows to a larger CSV file - including a translation for the collection-level record. Row 20, 100, and a couple others are translations, in addition to row 3. However, I couldn't get this CSV to import - it kept throwing the following error:

[info] [2019-08-16 13:17:20] Job 2003117 "arFileImportJob": Job started.
[info] [2019-08-16 13:17:20] Job 2003117 "arFileImportJob": Importing CSV file: large-items-collection.csv.
[info] [2019-08-16 13:17:20] Job 2003117 "arFileImportJob": Indexing imported records.
[info] [2019-08-16 13:17:20] Job 2003117 "arFileImportJob": Update type: import-as-new
[info] [2019-08-16 13:17:21] Job 2003117 "arFileImportJob": php '/usr/share/nginx/atom/symfony' 'csv:import' --index     --quiet --user-id="115215" --source-name='large-
items-collection.csv' '/usr/share/nginx/atom/uploads/tmp/TMPea7b5423.csv';   Call to undefined method QubitInformationObjectI18n::addProperty
[info] [2019-08-16 13:17:21] Job 2003117 "arFileImportJob": Job finished.

I have been using this CSV in 2.6 in previous testing (without the added translation rows) without any issue.

Thoughts?

#13 Updated by Mike Cantelon about 1 month ago

  • Status changed from Feedback to Code Review
  • Assignee deleted (Mike Cantelon)

#14 Updated by Steve Breker about 1 month ago

  • Status changed from Code Review to Feedback
  • Assignee set to Mike Cantelon

CR complete. Looks good.

#15 Updated by Mike Cantelon about 1 month ago

  • Status changed from Feedback to QA/Review
  • Assignee deleted (Mike Cantelon)

Merged into qa/2.6.x.

#16 Updated by Dan Gillean about 1 month ago

Mike, can you perhaps check and see if there is a bug with the language menu drop-down that appears on descriptions with translations? I'm still seeing some weirdness.

To reproduce:

  • Import the attached test sample, "rowlf-translation-test.csv"
  • This file contains a Fonds-level en description with an FR translation, and an Item-level en child record, that includes both an fr and es translation
  • After import, navigate to the new Fonds
  • Click on the Item level record
  • Open the language menu drop-down

Resulting error:

  • Only the French record is shown in the drop-down
  • If you use the global language menu to flip to es you can see the translation was imported - it's just not being indicated to the user via the language options drop-down

Otherwise, everything seemed to import well! I think if possible, we should try to resolve this issue, so that users are aware if multiple translations exist.

#17 Updated by Mike Cantelon about 1 month ago

Seems this happens when full-width treeview enabled. I've created a PR to fix.

#18 Updated by Mike Cantelon about 1 month ago

  • Status changed from Feedback to Code Review
  • Assignee deleted (Mike Cantelon)

#19 Updated by Mike Cantelon about 1 month ago

  • Status changed from Code Review to QA/Review
  • Assignee set to Dan Gillean

Fixed merged into qa/2.6.x.

#20 Updated by Dan Gillean about 1 month ago

  • Status changed from QA/Review to Feedback
  • Assignee changed from Dan Gillean to Mike Cantelon

Hi Mike!

Thanks, this resolved the issue with the language menu drop-down. Interesting to note it was related to full-width treeview and the AJAX page refresh partials.

The import itself seems to be behaving exactly as expected now. One minor issue I'm still noting is that the search/browse page will not show the records under the French language facet unless/until you run a full re-index separate from the import. I was importing via the user interface, which should be indexing progressively, and should make this need to fully reindex unnecessary. Any thoughts?

To reproduce

  • Log into AtoM and import the attached Rowlf test CSV
  • Confirm the import has completed successfully, then navigate to Browse > Archival descriptions
  • Apply the French facet under the Language facet heading

Resulting error

  • The French translation of the collection-level record for Rowlf (called "FRANCAIS Fonds de Rowlf") is not included in the results
  • Users must run a full re-index to see the collection-level translation

Expected result

  • When indexing progressively during an import, translations should be visible in the browse results when using the language facets immediately after import

#21 Updated by Mike Cantelon about 1 month ago

  • Status changed from Feedback to Code Review
  • Assignee deleted (Mike Cantelon)

#22 Updated by Mike Cantelon about 1 month ago

  • Status changed from Code Review to Feedback
  • Assignee set to Steve Breker

II'm updated the PR based on Steve's feedback.

#23 Updated by Steve Breker about 1 month ago

  • Assignee changed from Steve Breker to Mike Cantelon

CR complete. Looks good!

#24 Updated by Mike Cantelon about 1 month ago

  • Status changed from Feedback to QA/Review
  • Assignee deleted (Mike Cantelon)

Merged into qa/2.6.x.

#25 Updated by Dan Gillean about 1 month ago

  • Status changed from QA/Review to Verified

Hurrah! Thanks Mike! Looks good.

#26 Updated by Mike Cantelon 13 days ago

  • Status changed from Verified to Code Review

#27 Updated by Steve Breker 13 days ago

  • Status changed from Code Review to Feedback
  • Assignee set to Mike Cantelon

CR complete - looks good

Also available in: Atom PDF