Feature #13249

Add support for digital object CSV import/export for authority records

Added by Mike Cantelon 9 months ago. Updated 4 months ago.

Status:VerifiedStart date:06/27/2020
Priority:MediumDue date:
Assignee:Peter Van Garderen% Done:

100%

Category:Digital object
Target version:Release 2.6.0
Google Code Legacy ID: Tested version:2.5
Sponsored:Yes Requires documentation:No

Description

In Feature #12650 we added the ability to link a digital object to an authority record. However, at the time, adding digital object columns to the authority record CSV import and export templates was not included in the work completed for the AtoM 2.5 release.

This enhancement will add support for digital object import and export via CSV for authority records.

We will implement this work to match the existing method for including digital objects in an archival description CSV - that is, 2 new columns would be added to the authority record CSV template - digitalObjectUri, for external web-based digital objects, and digitalObjectPath, for importing local digital objects available on the server. Similar to the description export behavior, on export of an authority record CSV, links to the master digital object will be included in the digitalObjectUri column, using the base URL configured via Admin > Settings > Site information.

Screen Shot 2020-03-20 at 5.42.43 PM.png (349 KB) Peter Van Garderen, 03/20/2020 09:00 PM

Screen Shot 2020-04-03 at 12.35.33 PM.png (51.8 KB) Peter Van Garderen, 04/03/2020 03:41 PM


Subtasks

Bug #13370: Problem: the example authority records CSV file is missin...VerifiedPeter Van Garderen


Related issues

Related to Access to Memory (AtoM) - Feature #12650: Add the ability to upload a single digital object to an a... Verified 11/27/2018

History

#1 Updated by Dan Gillean 9 months ago

  • Related to Feature #12650: Add the ability to upload a single digital object to an authority record and supplement search/browse result stubs for authority records with thumbnails and history snippets added

#2 Updated by Dan Gillean 9 months ago

  • Description updated (diff)
  • Target version set to Release 2.6.0
  • Estimated time set to 12.00
  • Sponsored changed from No to Yes
  • Requires documentation set to Yes
  • Tested version 2.5 added

#3 Updated by Mike Cantelon 9 months ago

  • Status changed from New to Code Review
  • Assignee deleted (Mike Cantelon)

#4 Updated by Mike Cantelon 9 months ago

  • Status changed from Code Review to In progress
  • Assignee set to Mike Cantelon

#5 Updated by Mike Cantelon 9 months ago

  • Status changed from In progress to Code Review
  • Assignee deleted (Mike Cantelon)

#7 Updated by Mike Cantelon 8 months ago

  • Status changed from Code Review to QA/Review

Merged into qa/2.6.x.

#8 Updated by Peter Van Garderen 7 months ago

Feature: Add a digital object to an Authority record via CSV import
Given: A digital object is related but not connected to an AtoM Authority
Record
When: I use the Authority record CSV import template I can upload and link
an external digital object using a column for its web-based URI
And: I can upload and link a local digital object using a column for its
Filepath
Then: Authority records are saved with links to their digital objects

#9 Updated by Peter Van Garderen 7 months ago

    
    Feature: CSV export of digital objects linked to Authority records
    Given: An Authority record is linked to a digital object

    When: I use the Authority record CSV export template
    Then: The linked digital object master copy is included with each Authority 
        record in a column which contains its AtoM URI

#10 Updated by Peter Van Garderen 7 months ago

  • Assignee set to Peter Van Garderen

#11 Updated by Peter Van Garderen 7 months ago

The digital object CSV import via relative file path and external URL have both tested successfully on my Vagrant Ubuntu 18.04.3 box. When using the CLI CSV import you need to repopulate the search index for the new records to appear on the browse authority records page. When using the GUI CSV import the records appear but without their history snippet. Repopulating the search index fixes that.

For the sponsor client we need to re-test this feature on their AtoM Centos Testing VM as they will be running Centos in production. Therefore I will re-assign this ticket to Santi to follow-up. I can retest after this is setup.

#12 Updated by Peter Van Garderen 7 months ago

  • Assignee changed from Peter Van Garderen to Santiago Collazo

#13 Updated by Peter Van Garderen 7 months ago

  • Assignee changed from Santiago Collazo to Peter Van Garderen

#14 Updated by Peter Van Garderen 7 months ago

  • Assignee changed from Peter Van Garderen to Mike Cantelon

Hey Mike. As per my note above, I did run into this the first time I tested on Vagrant Ubuntu but assumed that manually updating the search index afterwards was just a standard requirement. From chatting with Radda I realize that's not the case and that the expected behaviour is for the search index to update automatically as part of the CSV import process.

I can confirm that running a CLI authority-record CSV import on Ubuntu does not automatically update the index. You have to manually run the php symfony search:populate CLI task for the imported authority records to show up on the Browse > Authority Records page. UPDATE: Using the --index flag when running the CLI task fixes this but the index update is still incomplete as reported for the GUI import below:

For the GUI authority-record CSV import it does appear to do some basic but incomplete indexing. The imported authority records do show up on the Browse > Authority Records page but is incomplete (the Authorized Form of Name shows incorrectly as "Untitled" and the history snippet and dates of existence are missing. If you follow the link to the detailed view page for that authority record this information is there (so it made it to the database) but it once again requires running the php symfony search:populate CLI task to update the Browse > Authority Records page correctly.

#15 Updated by Peter Van Garderen 7 months ago

The file path digital object upload is not working on CentOS for either the GUI or CLI import task. When using the Vagrant Ubuntu distro the CSV file path option works using /vagrant/file.jpg however on CentOS neither an absolute path (e.g. /usr/share/nginx/atom/csvimports/file.jpg) nor a relative path (e.g. csvimports/file.jpg) seem to be working.

#16 Updated by Mike Cantelon 7 months ago

I've figured out the indexing issue.

PR for CR: https://github.com/artefactual/atom/pull/1067

#17 Updated by Mike Cantelon 7 months ago

  • Assignee changed from Mike Cantelon to Peter Van Garderen

I did a test on the CENTOS test server where I added "tmp/goat.jpg" to the web directory, changed its owner/group to nginx, then modified the example authority record CSV (lib/task/import/example/authority_records/example_authority_records.csv) to pull it in and importing it as a local file worked for me.

http://nctr-atom.archivematica.net/example-person

So maybe it was a permissions issue causing problems?

#18 Updated by Mike Cantelon 7 months ago

(I've also updated the CENTOS server with the commit that fixes the authority import indexing issue.)

#19 Updated by Peter Van Garderen 7 months ago

I have QA'ed the indexing issue fix and can confirm that it works on both Ubuntu and CentOS. Thanks!

#20 Updated by Peter Van Garderen 7 months ago

Mike: I am still having trouble with digital object upload on the CentOS test server however. The first issue issue I appeared to have was using the correct absolute path. Using the tmp/ directory did fix that. However, I appear to still be having issues related to file permissions. I am able to include your sample tmp/goat.jpg file in a CSV import successfully. However, when I try to import any other file (after setting their owner/group to nginx -- see attached), the digital object is still not getting uploaded.

#21 Updated by Peter Van Garderen 7 months ago

  • Status changed from QA/Review to Document

So this CentOS issue has been diagnosed and resolved. If using 'tmp/' as the filepath in the CSV file then the digital objects need to be located in the 'usr/share/nginx/atom/tmp' directory and their ownership should be set to the 'nginx' user (which runs the AtoM application). GUI CSV import of DO works. However, when running the CLI csv:authority-import task it needs to be run with the command 'sudo -u nginx' to work. Otherwise the import task will import the metadata in the CSV but run into write permission issues for the file uploads ("File write to /usr/share/nginx/atom/uploads/r/null/... failed")

#22 Updated by Mike Cantelon 5 months ago

  • Assignee deleted (Mike Cantelon)

#23 Updated by Peter Van Garderen 5 months ago

  • Assignee set to Peter Van Garderen

#24 Updated by Peter Van Garderen 4 months ago

  • Status changed from Document to Verified

#26 Updated by Peter Van Garderen 4 months ago

  • Requires documentation changed from Yes to No

Also available in: Atom PDF