Feature #13280

Create relationships between existing authority records via CSV import

Added by Peter Van Garderen 7 months ago. Updated 4 months ago.

Status:VerifiedStart date:06/27/2020
Priority:HighDue date:
Assignee:Mike Cantelon% Done:

100%

Category:CSV import
Target version:Release 2.6.0
Google Code Legacy ID: Tested version:
Sponsored:Yes Requires documentation:No

Description

    Given: I import an authority records relationships CSV file

    When: An authority record in the Relationships CSV does not match an existing 
      authority record in the AtoM database 
    Then: Ignore that row in the Relationships CSV import file and don’t create 
      a new relationship record in the database
    And:  Provide a warning or log entry to notify the user that this relationship 
      was not created

    When: The import task is using the default “create new records” behaviour
    And: A newly imported relationship duplicates an existing relationship and 
     matching relationship type
    Then: Ignore that row in the relations CSV import file and don’t create a new 
      relationship record in the database
    And: Provide a warning or log entry to notify the user that this duplicate 
     relationship was not created

   When: The import task includes the “match and update” flag
   Then: Any existing relationships and matching relationship type between two authority records will not be 
      duplicated by the import

    When: The import task includes the “match and update” flag
    And:  The import data includes a relationship between authority records that already have an existing relationship (e.g. Alice is the child of Bob), but the relationship type is different
    Then: Create a new relationship between the two authority records using the new relationship type

    When: The import task includes the “match and update” flag
    And: The import data includes a relationship between authority records that already have an existing relationship (e.g. Alice is the child of Bob), but the relationship dates or description are different
   Then: Update the existing relationship with the new date or description information

Additional information: The current authority record relations CSV identifies actors by name with the “sourceAuthorizedFormOfName” and “targetAuthorizedFormOfName” fields. Authority record names do not need to be unique in AtoM though, so is simple name matching sufficient? Other options are matching existing AtoM authority records by slug, or doing authority record matches on multiple fields (e.g. name and institution)

Screen Shot 2020-06-02 at 6.09.02 PM.png (617 KB) Peter Van Garderen, 06/02/2020 09:11 PM


Subtasks

Bug #13369: CLI CSV import of example authority record with the examp...VerifiedPeter Van Garderen

Bug #13375: Problem: authority relations CSV import "--update=delete-...VerifiedMike Cantelon

Bug #13376: Problem: authority relations CSV import "--update=match-a...InvalidMike Cantelon

Bug #13377: Problem: warning is not provided when authority relations...VerifiedMike Cantelon

Bug #13378: Problem: authority relations CSV import "--update=match-a...InvalidPeter Van Garderen


Related issues

Related to Access to Memory (AtoM) - Feature #13277: Add multi-value alias names to the Authority record CSV i... Verified
Related to Access to Memory (AtoM) - Bug #13303: Problem: csv:authority-relation-import discontinues after... Verified 04/29/2020
Related to Access to Memory (AtoM) - Bug #13342: Problem: Adding Actor relationships via GUI is no longer ... Verified 06/03/2020

History

#1 Updated by Dan Gillean 7 months ago

  • Related to Feature #13277: Add multi-value alias names to the Authority record CSV import and export template added

#2 Updated by Peter Van Garderen 7 months ago

  • Subject changed from Create relationships with existing authority records via CSV import to Create relationships between existing authority records via CSV import

#3 Updated by Mike Cantelon 7 months ago

I think the second requirement ("An authority record in the Relationships CSV file does not match an existing authority record...") is moot now that we've separated relation import into a new task.

#4 Updated by Peter Van Garderen 6 months ago

Yes, Mike agree. I'll remove it from the requirement description. I'll also remove mention of the Authority Record CSV file from the Given statement.

#5 Updated by Peter Van Garderen 6 months ago

  • Description updated (diff)

#6 Updated by Peter Van Garderen 6 months ago

Note that the completed https://projects.artefactual.com/issues/13286 already implements this requirement:

When: The authority record relations CSV file includes reference to an authority record that already exists in the target AtoM instance
Then: A relationship with the matched authority record in the database is created

So it was removed from this Issue description.

#7 Updated by Peter Van Garderen 6 months ago

  • Description updated (diff)

#8 Updated by Mike Cantelon 6 months ago

  • Related to Bug #13303: Problem: csv:authority-relation-import discontinues after it encounters an invalid relationship type added

#9 Updated by Mike Cantelon 5 months ago

  • Status changed from New to Code Review
  • Assignee deleted (Mike Cantelon)

#10 Updated by Mike Cantelon 5 months ago

  • Status changed from Code Review to QA/Review
  • Assignee set to Peter Van Garderen

Merged to qa/2.6.x

Adds updating via either match and update or delete and replace.

#11 Updated by Peter Van Garderen 5 months ago

@Mike, these update behaviours are not present on the Import CSV GUI as import options. So I assume they are available as flags on the CLI CSV import? What is the correct syntax to use for these. I had no luck with the following (see screencap)

php symfony csv:authority-relation-import --match-and-update /path/to/file
php symfony csv:authority-relation-import --match /path/to/file
php symfony csv:authority-relation-import --update /path/to/file

#13 Updated by Mike Cantelon 5 months ago

You basically use them like this (kind of weird, but a precedent set in other updating CSV tasks):

--update="match-and-update" (updates existing actor)

and

--update="delete-and-replace" (delete actor then re-adds it)

#14 Updated by Mike Cantelon 5 months ago

  • Status changed from QA/Review to In progress
  • Assignee changed from Peter Van Garderen to Mike Cantelon

Change behavior to:

Delete and replace:
  • matches only on subject and object auth names
  • when a match is found, all existing relations between subject and object are deleted
  • multiple new relations between a subject and object are possible (it won't repeat the delete for every row and thereby delete the second relation in the csv)
Match-and-update:
  • matches on subject/object auth names and relationType

i.e. current behavior

#15 Updated by Peter Van Garderen 5 months ago

QA results:

When: An authority record in the Relationships CSV does not match an existing authority record in the AtoM database
[✓] Then: Ignore that row in the Relationships CSV import file and don’t create a new relationship record in the database
[✓] And: Provide a warning or log entry to notify the user that this relationship was not created

When: The import task is using the default “create new records” behaviour (i.e. no input flag)
And: A newly imported relationship duplicates an existing relationship and matching relationship type
[✓] Then: Ignore that row in the relations CSV import file and don’t create a new relationship record in the database
[X] And: Provide a warning or log entry to notify the user that this duplicate relationship was not created

When: The import task includes the “match and update” flag
[✓] Then: Any existing relationships and matching relationship type between two authority records will not be duplicated by the import

When: The import task includes the “match and update” flag
And: The import data includes a relationship between authority records that already have an existing relationship (e.g. Alice is the child of Bob), but the relationship type is different
[✓] Then: Create a new relationship between the two authority records using the new relationship type

When: The import task includes the “match and update” flag
And: The import data includes a relationship between authority records that already have an existing relationship (e.g. Alice is the child of Bob), but the relationship dates or description are different
[✓] Then: Update the existing relationship with the new date or description information

NEW:
When: The import task includes the “delete and replace” flag
And: The import data includes a relationship between authority records that already have relationships of one or more types.
[X] Then: All existing relationships between the two authority records are deleted.
[X] And: New relationship records are created using the relationship records in the CSV import file.

#16 Updated by Peter Van Garderen 5 months ago

Mike please note that, in addition to the updated requirements for the "delete-and-replace" flag, QA also found one small omission in this feature:

When: The import task is using the default “create new records” behaviour (i.e. no input flag)
And: A newly imported relationship duplicates an existing relationship and matching relationship type
[✓] Then: Ignore that row in the relations CSV import file and don’t create a new relationship record in the database
[X] And: Provide a warning or log entry to notify the user that this duplicate relationship was not created 

We are currently not getting that warning that a duplicate relationship was encountered but not created.

#17 Updated by Peter Van Garderen 4 months ago

  • Related to Bug #13342: Problem: Adding Actor relationships via GUI is no longer working added

#18 Updated by Peter Van Garderen 4 months ago

  • Requires documentation changed from Yes to No

#19 Updated by Dan Gillean 4 months ago

  • Status changed from In progress to Verified

Also available in: Atom PDF