Feature #10853

Add more fields to repository CSV import

Added by Dan Gillean over 3 years ago. Updated over 2 years ago.

Status:VerifiedStart date:02/02/2017
Priority:MediumDue date:
Assignee:-% Done:

0%

Category:CSV importEstimated time:1.00 hour
Target version:Release 2.4.1
Google Code Legacy ID: Tested version:
Sponsored:No Requires documentation:

Description

In the current version added, some basic fields are missing. This feature ticket will incorporate some of the easier fields to add, including:

DESCRIPTION AREA
  • Administrative structure (ISDIAH 5.3.4)
  • Records management and collecting policies (ISDIAH 5.3.5)
ACCESS AREA
  • Conditions and requirements (ISDIAH 5.4.2)
  • Accessibility (ISDIAH 5.4.3)
SERVICES AREA
  • Research services (ISDIAH 5.5.1)
  • Reproduction services (ISDIAH 5.5.2)
  • Public areas (ISDIAH 5.5.3)
CONTROL AREA
  • Institution identifier (ISDIAH 5.6.2)
  • Dates of creation, revision and deletion (ISDIAH 5.6.6)
  • Sources (ISDIAH 5.6.8)

This does NOT cover all missing fields, and futher work should be done to include missing fields such as History, Other forms of name, address fields such as City, Region, and Country, etc.

example-repositories.csv Magnifier - Updated CSV example for use in lib/task/import/example (7.68 KB) Dan Gillean, 01/03/2018 04:31 PM


Related issues

Related to Access to Memory (AtoM) - Feature #10472: Add repository CSV import via UI and job scheduler Verified 10/25/2016
Related to Access to Memory (AtoM) - Feature #8591: Create an export:repositories task Verified 06/22/2015
Related to Access to Memory (AtoM) - Bug #11842: Clipboard export of repositories fails Verified 01/03/2018
Blocked by Access to Memory (AtoM) - Bug #11654: Clipboard not behaving consistently as user navigates Invalid 10/27/2017

History

#1 Updated by Dan Gillean over 3 years ago

  • Description updated (diff)
  • Category set to CSV import
  • Status changed from New to Code Review
  • Assignee changed from Dan Gillean to Mike Cantelon
  • Target version set to Release 2.4.0
  • Estimated time set to 1.00
  • Requires documentation set to Yes

Mike, can you please code review this for me?

https://github.com/artefactual/atom/pull/513

#2 Updated by Dan Gillean over 3 years ago

  • File example_repositories.csv added
  • Status changed from Code Review to Verified
  • Assignee deleted (Mike Cantelon)

Merged!

Attaching a copy of the updated CSV example for reference.

Would love to talk over with a developer how to add some of the more complicated fields. History field and the missing contact fields (city, region, country) are top priority.

Thanks!

#3 Updated by Dan Gillean over 3 years ago

  • Related to Feature #10472: Add repository CSV import via UI and job scheduler added

#4 Updated by Dan Gillean over 3 years ago

  • Status changed from Verified to Feedback

In ticket #10015, we have now added our first pass at repository export. With the hope of revisiting this ticket in the future, i'm adding notes on things that export that do not currently import, and/or are not included on the repository import example CSV.

parallelFormsOfName - Parallel form(s) of name (ISDIAH 5.1.3): "Record the parallel form(s) of name of the institution in accordance with any relevant national or international conventions or rules applied by the agency that created the description, including any necessary sub elements and/or qualifiers required by those conventions or rules. Speiy in the Rules and/or conventions used element (5.6.3) which rules have been applied."

otherFormsOfName - Other form(s) of name (ISDIAH 5.1.4): "Record any other name(s) by which the institution may be known. This could include other forms of the same name, acronyms, other institutional names, or changes of name over time, including, if possible, relevant dates."

types - [access point for institution type in Identity area. Accepts piped values]

history - History (ISDIAH 5.3.1): "Record any relevant information about the history of the institution. This element may include information on dates of establishment, changes of names, changes of legislative mandates, or any other sources of authority for the institution."

  • Imported okay on first test, but is not present on the CSV import example.

mandates: Mandates/Sources of authority (ISDIAH 5.3.3): "Record any document, law, directive or charter which acts as a source of authority for the powers, functions and repsonibilities of the institution, together with information on the juridiction(s) and convering dates when the mandate(s) applied or were changed."

  • Imported okay on first test, but is not present on the CSV import example.

buildings - Buildings (ISDIAH 5.3.6): "Record information on the building(s) of the institution (general and architectural characteristics of the building, capacity of storage areas, etc). Where possible, provide information which can be used for generating statistics."

descIdentitifer - Description identifier (ISDIAH 5.6.1): "Record a unique identifier in accordance with local and/or national conventions. If the description is to be used internationally, record the code of the country in which the description was created in accordance with the latest version of ISO 3166 - Codes for the representation of names of countries. Where the creator of the description is an international organisation, give the organisational identifier in place of the country code."

  • Imported okay on first test, but is not present on the CSV import example.

descRules - Rules and/or conventions used (ISDIAH 5.6.3): "Record the names, and, where useful, the editions or publication dates of the conventions or rules applied. Speficy, separately, which rules have been applied for creating the Authorised form(s) of name. Include reference to any system(s) of dating used to identify dates in this description (e.g. ISO 8601)."

  • Imported okay on first test, but is not present on the CSV import example.

languages - [autocomplete for languages in Control area. Accepts piped values]

scripts - [autocomplete for scripts in Control area. Accepts piped values]

thematicAreas - [access point for thematic areas. Accepts piped values]

geographicSubregions - [access point for geographic subregions. Accepts piped values.]

#5 Updated by Dan Gillean over 3 years ago

Note as well that the remaining key contact fields (city, region, country) are missing on both import and export

#6 Updated by Dan Gillean over 3 years ago

  • Related to Feature #8591: Create an export:repositories task added

#7 Updated by Dan Gillean over 3 years ago

  • Target version deleted (Release 2.4.0)
  • Requires documentation deleted (Yes)

The iniitial enhancements I've added are in 2.4, but since that is the first time the repository csv import/export appears, and because there is still much to do to make sure all fields can import/export, I'm removing the version from this. Notes have been added to the documentation about what fields work/don't work for import.

#8 Updated by Nick Wilkinson about 3 years ago

  • Assignee set to Mike Cantelon

Hi Mike, can you please look into this?

#9 Updated by Dan Gillean about 3 years ago

  • File example-repositories-2.4-TODO.csv added
  • File example_repositories-2.4.csv added

Mike I'm adding a new version of the CSV, where all columns we need are present. the TODO CSV i'm attaching is where I have added the missing contact area fields we need - I've made up the column header names for these based on guesses as to what they should be.

I'm also adding a version withOUT the columns I added based on guesses, for testing: Note that some of the other fields in here don't currently import, and/or don't export. Run a test with this to see what works and what doesn't. The goal of this ticket is so we can roundtrip all repository data.

We'll also need to make sure that the sample is updated. in lib/task/import/example when we have everything working.

#10 Updated by Mike Cantelon about 3 years ago

Sounds good... thanks guys.

#11 Updated by Mike Cantelon about 3 years ago

  • Status changed from Feedback to Code Review
  • Assignee changed from Mike Cantelon to Nick Wilkinson

#12 Updated by Mike Cantelon about 3 years ago

  • Assignee changed from Nick Wilkinson to Mike Cantelon

Actually I'm going to do a bit more refactoring before creating the PR.

#13 Updated by Mike Cantelon about 3 years ago

  • Assignee changed from Mike Cantelon to Nick Wilkinson

#14 Updated by Nick Wilkinson about 3 years ago

  • Assignee changed from Nick Wilkinson to Steve Breker

Hi Steve, assigning to you for CR.

#15 Updated by Steve Breker about 3 years ago

  • Status changed from Code Review to Feedback
  • Assignee changed from Steve Breker to Mike Cantelon

Big change! I've read this PR over - no changes to recommend. Looks good to me.

#16 Updated by Mike Cantelon about 3 years ago

  • Status changed from Feedback to QA/Review
  • Assignee changed from Mike Cantelon to Dan Gillean

Merged and ready for QA.

#17 Updated by Dan Gillean almost 3 years ago

  • Blocked by Bug #11654: Clipboard not behaving consistently as user navigates added

#18 Updated by Dan Gillean almost 3 years ago

  • Status changed from QA/Review to Feedback
  • Assignee changed from Dan Gillean to Mike Cantelon

Hi Mike,

Very very close now!

We're just missing some important fields in the contact area. These are:

  • region
  • country
  • postal code
  • city

All are free-text fields, except for country, which links to a controlled list of countries provided by Symfony (I think - they are not in an AtoM taxonomy). I think that ideally it should match text input from the user (case insensitive), and if no match is found, skip the field without making the whole import fail, perhaps outputting a warning in the console.

There are other fields in the contact area, but these 4 are the critical ones. We need:
  • Support for them on import
  • Support for them on CSV export
  • Them to be added to the example repositories CSV in lib/task/import/example. I can populate the fields with sample data if you give me a version of the current example with the additional fields added.

Once we have this verified, we are also going to want to backport it to stable/2.4.x, for inclusion in 2.4.1.

#19 Updated by Redmine Admin almost 3 years ago

  • Status changed from Feedback to Code Review
  • Assignee changed from Mike Cantelon to Nick Wilkinson

#20 Updated by Nick Wilkinson almost 3 years ago

  • Assignee changed from Nick Wilkinson to Steve Breker

Hi Steve, can you please take a look at this again for CR?

#21 Updated by Steve Breker almost 3 years ago

  • Status changed from Code Review to Feedback
  • Assignee changed from Steve Breker to Nick Wilkinson

Code review complete - looks good.

#22 Updated by Nick Wilkinson almost 3 years ago

  • Assignee changed from Nick Wilkinson to Mike Cantelon

Hi Mike, do you need to merge this or is it ready for QA?

#23 Updated by Mike Cantelon almost 3 years ago

I'll need to merge it, but should be quick (I'll do that right now then will update).

#24 Updated by Mike Cantelon almost 3 years ago

  • Assignee changed from Mike Cantelon to Dan Gillean

I updated the PR with updated example CSV (I wasn't sure yesterday how to handle passing this off to Dan, but committing the work to the branch is probably the right way):

lib/task/import/example/example_repositories.csv

So, Dan, feel free to update the sample values I threw in for the new CSV columns then once you're done I'll merge it.

Also, before I forget, a note for QAing this: maybe test the accessions import as well as it now shares helper code with the repository import (code for normalizing country name or code to country code).

#25 Updated by Dan Gillean almost 3 years ago

  • File example-repositories.csvMagnifier added
  • Assignee changed from Dan Gillean to Mike Cantelon
  • Target version set to Release 2.4.1

Looks good on import! I made a few minor changes in the sample data, and have attached an updated issue.

I couldn't really test export - something is up with the clipboard, and i'm not sure if it's just this branch or 2.5 in general, but:

  • Add the 2 repositories to the clipboard
  • Navigate to clipboard
  • Flip to Archival institution, and hit export

Resulting error: Message stating: "Export complete - no records were exported."

Expected result: Clipboard repository results are exported in CSV format.

I can file a new issue for this if needed but can you make sure the branch has been rebased against 2.5 recently? CLI repo export works fine.

#26 Updated by Mike Cantelon almost 3 years ago

Cool. Okay, I'll update with your CSV file and will merge.

The branch was rebased with qa/2.5.x before I made the PR, yeah, so filing a new issue's probably best. I'll try to replicate what you ran into... hopefully it'll be an easy fix.

#27 Updated by Mike Cantelon almost 3 years ago

  • Status changed from Feedback to QA/Review
  • Assignee changed from Mike Cantelon to Dan Gillean

I've merged this to qa/2.5.x.

#28 Updated by Mike Cantelon almost 3 years ago

I think I know what the issue you ran into is... I'll make an issue for it and will relate it to this one.

EDIT: #11842 is the issue.

#29 Updated by Mike Cantelon almost 3 years ago

  • Related to Bug #11842: Clipboard export of repositories fails added

#30 Updated by Mike Cantelon almost 3 years ago

I've fixed that related issue in qa/2.5.x.

#31 Updated by Dan Gillean almost 3 years ago

  • Assignee changed from Dan Gillean to Mike Cantelon

Looks good, Mike - we just need to do a couple things.

1) Make sure the final updated version of the CSV (attached in note25, above) is added to lib/task/import/example if it hasn't already been
2) Backport this to the stable/2.4.x, so it can be included in 2.4.1. I'm not sure if we need to do that with issue #11842 as well or not....

Thanks

#32 Updated by Mike Cantelon almost 3 years ago

  • Assignee changed from Mike Cantelon to Dan Gillean

Thanks Dan... I've backported this and #11842 to stable/2.4.x. Both 2.4 and 2.5 branches should have the updated CSV example file.

#33 Updated by Dan Gillean over 2 years ago

  • File deleted (example_repositories-2.4.csv)

#34 Updated by Dan Gillean over 2 years ago

  • File deleted (example-repositories-2.4-TODO.csv)

#35 Updated by Dan Gillean over 2 years ago

  • File deleted (example_repositories.csv)

#36 Updated by Dan Gillean over 2 years ago

  • Status changed from QA/Review to Feedback
  • Assignee changed from Dan Gillean to Mike Cantelon

Unfortunately a whole bunch of these fields do not import properly in 2.4.1 testing. These include:

  • all alternative forms of name fields, and type in the Identity Area
  • the Accessibility field in the Access Area
  • the entire Control area
  • both additional access points

Full list of CSV fields that did not import in 2.4.1:

  • paralellFormsOfName
  • otherFormsOfName
  • types
  • disabledAccess
  • descriptionIdentifier
  • institutionIdentifier
  • descriptionRules
  • descriptionStatus
  • levelOfDetail
  • desrciptionRevisionHistory
  • language
  • script
  • maintenanceNote
  • geographicSubregions
  • thematicAreas

Tested both a GUI and a CLI import, in the vagrant box.

#37 Updated by Mike Cantelon over 2 years ago

  • Status changed from Feedback to Code Review
  • Assignee changed from Mike Cantelon to Nick Wilkinson

I'd missed some functionality when backporting.

I'm not sure if this needs to be code reviewed given it's 2.5 code, but just in case here's a PR for CR: https://github.com/artefactual/atom/pull/656

#38 Updated by Nick Wilkinson over 2 years ago

  • Assignee changed from Nick Wilkinson to Steve Breker

Hi Steve, passing to you for CR.

#39 Updated by Steve Breker over 2 years ago

  • Status changed from Code Review to Feedback
  • Assignee changed from Steve Breker to Mike Cantelon

Code review complete. Looks good.

#40 Updated by Mike Cantelon over 2 years ago

  • Status changed from Feedback to QA/Review
  • Assignee changed from Mike Cantelon to Dan Gillean

Merged backported fix to stable/2.4.x.

#41 Updated by Dan Gillean over 2 years ago

  • Status changed from QA/Review to Verified
  • Assignee deleted (Dan Gillean)

Hurray! Thanks, Mike!

Also available in: Atom PDF