Bug #13067

DC metadata not imported into AtoM when DIP is created from a transfer placed in backlog

Added by Dan Gillean 4 months ago. Updated 29 days ago.

Status:VerifiedStart date:05/29/2019
Priority:HighDue date:
Assignee:Sarah Romkey% Done:

0%

Category:Archivematica integration
Target version:Release 2.5.2
Google Code Legacy ID: Tested version:2.4
Sponsored:No Requires documentation:

Description

Issue reported by Evelyn - tested in: AM 1.9.1, AtoM 2.4.1

To reproduce

  • In Archivematica, create a transfer and place it in transfer backlog
  • In the Appraisal Tab, retrieve the transfer and create a SIP from it
  • Enter DC metadata via the GUI
  • Create a DIP and upload it to AtoM

Error encountered

No Information Object is created and none of the DC metadata appear in AtoM.

Note that the error may be caused by the presence of a second structMap in the METS file when a transfer is placed in backlog. This is a logical structMap with LABEL="Hierarchical". For transfers run straight through Archivematica without being placed in backlog, there is only the default physical structMap. It's possible that AtoM doesn't know what to do with the fact that the dmdSec is referenced from two structMaps.

Expected result

AtoM should create an information object out of the Dublin Core metadata entered via the GUI in Archivmatica.

Notes

See the related AM issue and conversation for more details: https://github.com/archivematica/Issues/issues/570

One of the AM team said this might be related: https://github.com/archivematica/Issues/issues/586

Radda noted the following:

Probably the PREMIS 3 upgrade requires changes in AtoM's METS parsing process ...

https://github.com/artefactual/atom/blob/qa/2.6.x/lib/QubitMetsParser.class.php#L31


Related issues

Related to Access to Memory (AtoM) - Bug #13068: AtoM not creating Information Object for re-ingested AIPs... Verified 05/29/2019

History

#1 Updated by Evelyn McLellan 4 months ago

I don't think https://github.com/archivematica/Issues/issues/586 is related to this. That issue has to do with the transfer METS file, which doesn't affect DIP upload to AtoM.

#2 Updated by Evelyn McLellan 4 months ago

  • Subject changed from DC metadata not imported into AtoM when METS file has logical structMap to DC metadata not imported into AtoM when DIP is created from a transfer placed in backlog

#3 Updated by José Raddaoui Marín 4 months ago

My note about the PREMIS 3 upgrade was thinking that this issue happened using the "qa" branches of AM (to be 1.10), not using the 1.9.1 version, so that's probably not the issue in this case.

Nevertheless, we should take in consideration the PREMIS upgrade too, to avoid issues in the future, as we're using the version 2 namespace in several places:

https://github.com/artefactual/atom/blob/qa/2.6.x/lib/QubitMetsParser.class.php (look for: info:lc/xmlns/premis-v2)

#4 Updated by Dan Gillean 4 months ago

  • Related to Bug #13068: AtoM not creating Information Object for re-ingested AIPs with updated metadata added

#5 Updated by José Raddaoui Marín about 1 month ago

Hi Dan and Evelyn,

Fixing the PREMIS issue solved most of the problems with the hierarchical DIP upload on the AtoM side. However, AtoM only creates an intermediate archival description from the Dublin Core metadata entered via the GUI in Archivematica on the non hierarchical DIP upload.

When the import is performed from the hierarchical structMap, the "objects" DMD section is not considered and AtoM only uses the TYPE attribute from its structMap div to update the level of description of the target description. This seems to have been always like this in the code but I could not find an explanation in the documentation. We could easily change that behavior if you want but I'm not sure if that's something we want to do.

Please, let me know what do you think about it.

#6 Updated by José Raddaoui Marín about 1 month ago

  • Status changed from New to In progress
  • Assignee set to José Raddaoui Marín
  • Target version set to Release 2.5.2

#7 Updated by Sarah Romkey about 1 month ago

I'm not totally sure I follow... is the remaining problem that you can't combine the hierarchical structMap workflow with having DC metadata entered via the Archivematica GUI? If that is the case, I would consider it an edge case and not worry about a fix at this moment.

#8 Updated by Sarah Romkey about 1 month ago

Edit- I think I understand this now.

My understanding is that the "top" description created in the hierarchy gets assigned to the objects directory in the structMap. Ideally, the DC metadata would also get assigned to the objects folder and displayed at that level in AtoM. However, if that's not currently supported functionality, it's not a crucial fix at this moment IMO.

Radda, let me know if that helps!!

#9 Updated by Dan Gillean about 1 month ago

  • Status changed from In progress to Feedback

Hi Radda and Evelyn,

After some discussion with the AM team, mostly Sarah Romkey, we've decided to leave this as is for now. Sarah thinks that creating a new intermediary description to capture the aggregate metadata may be confusing/frustrating to users. However, we can't overwrite the existing metadata that may be on the target description either... so for now, Sarah thinks we should just ignore the aggregate DC metadata from Archivemtatica's GUI DC template, and they will add a note about this in their documentation. In the future we can think of a better solution, but right now it feels like there are a number of edge cases that might require more consideration.

So: that part of the ticket can be marked as invalid for now! Not sure if there are other parts that mean we don't want to mark this whole ticket as such or not.

#10 Updated by José Raddaoui Marín about 1 month ago

Thanks Sarah and Dan,

I'd leave this open to verify that the hierarchical DIP upload works, as it was not creating any archival description before the fix. It should create descriptions now, except for the intermediate level with the metadata from the GUI. I could not test the directories creation due to an AM issue that seems to be fixed now, so I'll give it a final try with that fix before marking this for code review.

#11 Updated by José Raddaoui Marín about 1 month ago

  • Status changed from Feedback to Code Review

Hi Sarah and Evelyn,

I've tested the AtoM fixes with Douglas' changes on the AM side and everything seems to be working as expected. However, I went a bit further and tested a partial re-ingest from an arranged SIP and what I noticed is that the logical structMap is present on the first METS but missing on the re-ingested one. Is that expected or should the logical structMap be maintained after the re-ingest?

Besides that (which would be an issue on the AM side), this is ready for code review:

https://github.com/artefactual/atom/pull/951

#12 Updated by José Raddaoui Marín about 1 month ago

  • Assignee changed from José Raddaoui Marín to Douglas Cerna

#13 Updated by Douglas Cerna about 1 month ago

CR complete. Code looks great to me!

#14 Updated by José Raddaoui Marín about 1 month ago

  • Status changed from Code Review to QA/Review
  • Assignee changed from Douglas Cerna to Dan Gillean

Thanks Douglas!

Ready for testing in qa/2.6.x and dev/dip-upload-stable (based on stable/2.5.x with all the DIP upload related fixes).

#15 Updated by Sara Allain 30 days ago

  • Assignee changed from Dan Gillean to Sarah Romkey

The items appear in AtoM with object-level metadata. Aggregate metadata (added via the GUI in Archivematica) is not uploaded to AtoM.

However, while testing this I noticed that on upload the DIP loses its hierarchical-ness. I created a two-level arrangement:

arranged/
└── objects
    ├── Landing_zone.jpg
    ├── level-2/
    │   ├── bird.mp3
    │   └── MARBLES.TGA
    └── View_from_lookout_over_Queenstown_towards_the_Remarkables_in_spring.jpg

The upload flattened the structure (see https://atom25.1804.qa.accesstomemory.net/weve-got-metadata). I think this is a bug, but I'll pass it on to Sarah Romkey for her opinion.

The hierarchical logical structmap is present in the AIP METS.

#16 Updated by José Raddaoui Marín 29 days ago

Hi Sara(h)s,

I thought that was an issue too, but I found a note about it in the documentation (the tip at the bottom).

#17 Updated by Sara Allain 29 days ago

  • Status changed from QA/Review to Verified

Ah okay! Thanks Radda! We can call this verified then :)

Also available in: Atom PDF