Feature #9556

Add csv:import option to skip digital object derivative creation

Added by David Juhasz about 6 years ago. Updated almost 5 years ago.

Status:VerifiedStart date:03/11/2016
Priority:MediumDue date:
Assignee:Dan Gillean% Done:

0%

Category:CSV import
Target version:Release 2.3.0
Google Code Legacy ID: Tested version:
Sponsored:Yes Requires documentation:

Description

Add a flag to the csv:import CLI task to skip creation of digital object derivatives. This can be desirable to reduce the overall memory usage or time of the import task - derivatives can be created later via the digitalobject:regen-derivatives task.

In the case of remote digital object (digitalPathURI) this will prevent downloading the digital objects during the import as well, reducing the memory requirements and time required for the import even further.

example_information_objects_isad.csv Magnifier - Test file for CSV import (4.07 KB) David Juhasz, 04/21/2016 04:54 PM

History

#4 Updated by Dan Gillean about 6 years ago

  • Sponsored changed from No to Yes
  • Requires documentation set to Yes

#5 Updated by Mike Cantelon about 6 years ago

Reviewed the PR and the code looks good!

#6 Updated by David Juhasz about 6 years ago

N.B.: there's a subtle difference in how bad URIs are handled when the --skip-derivatives option.

  1. If "--skip-derivatives" is used, then the URI will be written to the database, and the URI will be displayed in the "Archival description > Digital object metadata" section (default settings), even if it doesn't resolve (e.g. there's a typo).
  2. If not --skip-derivatives, and the URI doesn't resolve then no digital object URI will be written to the database, and no digital object metadata will be linked to the archival description.

In the first case we aren't trying to resolve the URI, so we don't know if it's bad. In the second case I don't think we want to store a bad URI.

#7 Updated by David Juhasz about 6 years ago

  • File example_information_objects_isad.csv added

Attach CSV file used for testing.

#8 Updated by David Juhasz about 6 years ago

  • Status changed from New to Code Review
  • Assignee changed from David Juhasz to Jesús García Crespo

Hi Jesús,

I added some additional changes to PR #313. Can you please CR?

Thanks,
David

#9 Updated by Jesús García Crespo about 6 years ago

  • Status changed from Code Review to In progress

Done, one minor observation but LGTM.

#10 Updated by Jesús García Crespo about 6 years ago

  • Assignee changed from Jesús García Crespo to David Juhasz

#11 Updated by David Juhasz about 6 years ago

  • Status changed from In progress to Code Review
  • Assignee changed from David Juhasz to Jesús García Crespo

One last quick CR please Jesús: I overrode the sfTask::log() class to give me a single method for logging and echoing a message.

#12 Updated by David Juhasz about 6 years ago

  • File deleted (example_information_objects_isad.csv)

#13 Updated by David Juhasz about 6 years ago

Replace previous, bad, csv test file

#14 Updated by Jesús García Crespo about 6 years ago

  • Status changed from Code Review to In progress
  • Assignee changed from Jesús García Crespo to David Juhasz

#15 Updated by David Juhasz about 6 years ago

Squashed and merged to qa/2.3.x

#16 Updated by David Juhasz about 6 years ago

  • Status changed from In progress to Verified

#17 Updated by David Juhasz about 6 years ago

  • Status changed from Verified to Document
  • Assignee changed from David Juhasz to Dan Gillean

#18 Updated by Dan Gillean about 6 years ago

  • Status changed from Document to Verified

Documentation still to come - bu we are using the "requires documentation" flag to track this, so we can verify the issue.

#19 Updated by Dan Gillean almost 5 years ago

  • Requires documentation deleted (Yes)

Also available in: Atom PDF