Add support for digital object CSV import/export for authority records
|Assignee:||Peter Van Garderen||% Done:|
|Target version:||Release 2.6.0|
|Google Code Legacy ID:||Tested version:||2.5|
In Feature #12650 we added the ability to link a digital object to an authority record. However, at the time, adding digital object columns to the authority record CSV import and export templates was not included in the work completed for the AtoM 2.5 release.
This enhancement will add support for digital object import and export via CSV for authority records.
We will implement this work to match the existing method for including digital objects in an archival description CSV - that is, 2 new columns would be added to the authority record CSV template -
digitalObjectUri, for external web-based digital objects, and
digitalObjectPath, for importing local digital objects available on the server. Similar to the description export behavior, on export of an authority record CSV, links to the master digital object will be included in the
digitalObjectUri column, using the base URL configured via Admin > Settings > Site information.
#8 Updated by Peter Van Garderen 7 months ago
Feature: Add a digital object to an Authority record via CSV import
Given: A digital object is related but not connected to an AtoM Authority
When: I use the Authority record CSV import template I can upload and link
an external digital object using a column for its web-based URI
And: I can upload and link a local digital object using a column for its
Then: Authority records are saved with links to their digital objects
#9 Updated by Peter Van Garderen 7 months ago
Feature: CSV export of digital objects linked to Authority records Given: An Authority record is linked to a digital object When: I use the Authority record CSV export template Then: The linked digital object master copy is included with each Authority record in a column which contains its AtoM URI
#11 Updated by Peter Van Garderen 7 months ago
- File Screen Shot 2020-03-20 at 5.42.43 PM.png added
- Priority changed from Medium to High
The digital object CSV import via relative file path and external URL have both tested successfully on my Vagrant Ubuntu 18.04.3 box. When using the CLI CSV import you need to repopulate the search index for the new records to appear on the browse authority records page. When using the GUI CSV import the records appear but without their history snippet. Repopulating the search index fixes that.
For the sponsor client we need to re-test this feature on their AtoM Centos Testing VM as they will be running Centos in production. Therefore I will re-assign this ticket to Santi to follow-up. I can retest after this is setup.
#14 Updated by Peter Van Garderen 7 months ago
- Assignee changed from Peter Van Garderen to Mike Cantelon
Hey Mike. As per my note above, I did run into this the first time I tested on Vagrant Ubuntu but assumed that manually updating the search index afterwards was just a standard requirement. From chatting with Radda I realize that's not the case and that the expected behaviour is for the search index to update automatically as part of the CSV import process.
I can confirm that running a CLI authority-record CSV import on Ubuntu does not automatically update the index. You have to manually run the php symfony search:populate CLI task for the imported authority records to show up on the Browse > Authority Records page. UPDATE: Using the --index flag when running the CLI task fixes this but the index update is still incomplete as reported for the GUI import below:
For the GUI authority-record CSV import it does appear to do some basic but incomplete indexing. The imported authority records do show up on the Browse > Authority Records page but is incomplete (the Authorized Form of Name shows incorrectly as "Untitled" and the history snippet and dates of existence are missing. If you follow the link to the detailed view page for that authority record this information is there (so it made it to the database) but it once again requires running the php symfony search:populate CLI task to update the Browse > Authority Records page correctly.
#15 Updated by Peter Van Garderen 7 months ago
The file path digital object upload is not working on CentOS for either the GUI or CLI import task. When using the Vagrant Ubuntu distro the CSV file path option works using /vagrant/file.jpg however on CentOS neither an absolute path (e.g. /usr/share/nginx/atom/csvimports/file.jpg) nor a relative path (e.g. csvimports/file.jpg) seem to be working.
#17 Updated by Mike Cantelon 7 months ago
- Assignee changed from Mike Cantelon to Peter Van Garderen
I did a test on the CENTOS test server where I added "tmp/goat.jpg" to the web directory, changed its owner/group to nginx, then modified the example authority record CSV (
lib/task/import/example/authority_records/example_authority_records.csv) to pull it in and importing it as a local file worked for me.
So maybe it was a permissions issue causing problems?
#20 Updated by Peter Van Garderen 7 months ago
- File Screen Shot 2020-04-03 at 12.35.33 PM.png added
- Assignee changed from Peter Van Garderen to Mike Cantelon
Mike: I am still having trouble with digital object upload on the CentOS test server however. The first issue issue I appeared to have was using the correct absolute path. Using the tmp/ directory did fix that. However, I appear to still be having issues related to file permissions. I am able to include your sample tmp/goat.jpg file in a CSV import successfully. However, when I try to import any other file (after setting their owner/group to nginx -- see attached), the digital object is still not getting uploaded.
#21 Updated by Peter Van Garderen 7 months ago
- Status changed from QA/Review to Document
So this CentOS issue has been diagnosed and resolved. If using 'tmp/' as the filepath in the CSV file then the digital objects need to be located in the 'usr/share/nginx/atom/tmp' directory and their ownership should be set to the 'nginx' user (which runs the AtoM application). GUI CSV import of DO works. However, when running the CLI csv:authority-import task it needs to be run with the command 'sudo -u nginx' to work. Otherwise the import task will import the metadata in the CSV but run into write permission issues for the file uploads ("File write to /usr/share/nginx/atom/uploads/r/null/... failed")