Bug #11213

atom dip upload not working

Added by Nick Wilkinson over 2 years ago. Updated about 1 month ago.

Status:VerifiedStart date:05/31/2017
Priority:CriticalDue date:
Assignee:Sarah Romkey% Done:

0%

Category:Access
Target version:Release 1.7.0
Google Code Legacy ID: Pull Request:
Sponsored:No Requires documentation:

Description

As described in https://github.com/artefactual/archivematica/issues/622

The atom dip upload feature is not working in qa/1.x

  • When using the web interface, the question "Upload DIP" ->"Upload DIP to Atom" asks for the atom slug, but never finishes and the question is always in "Awaiting decision" state.
  • Using am mcp-rpc-cli, when selecting "Upload DIP to Atom" for the "Upload DIP" question, there is a new question, with only one choice ( upload-qubit_v0.0):
    <choicesAvailableForUnit> <UUID>3f857f46-6a10-472d-a564-77bd7534f58b</UUID> <unit> <type>DIP</type> <unitXML> <UUID>c8529576-a069-48fc-b056-8e260275e86c</UUID> <currentPath>%sharedPath%watchedDirectories/uploadDIP/atom4-c8529576-a069-48fc-b056-8e260275e86c/</currentPath> </unitXML> </unit> <choices> <choice> <chainAvailable>0</chainAvailable> <description>upload-qubit_v0.0</description> </choice> </choices> </choicesAvailableForUnit>
    
  • The switch --rsync-command="None"\ is being passed to the script, and that makes rsync fail with error 14

Related issues

Related to Access to Memory (AtoM) - Bug #11721: DIP upload logic incompatible with METS logical structMap Verified 11/21/2017
Related to Archivematica - Feature #11792: Give the user a choice, during AIP creation, whether or n... Deploy 12/10/2017

History

#1 Updated by Kelly Stewart almost 2 years ago

  • Assignee set to Nick Wilkinson
  • Priority changed from Medium to High

#2 Updated by Nick Wilkinson almost 2 years ago

  • Assignee changed from Nick Wilkinson to Mike Cantelon
  • Priority changed from High to Critical

Hi Mike, assigning to you to investigate. This should be considered a top priority. Please work closely with Kelly on this for more background on the bug and where you can deploy this so she can test again.

#3 Updated by Mike Cantelon almost 2 years ago

  • Status changed from New to Code Review
  • Assignee changed from Mike Cantelon to Nick Wilkinson

PR for CR: https://github.com/artefactual/archivematica/pull/795

There's still an issue with DIP upload that I'm trying to fix, but it's a separate issue and, I'd guess, might get resolved simply by re-deploying.

#4 Updated by Nick Wilkinson almost 2 years ago

  • Assignee changed from Nick Wilkinson to Joel Dunham

Hi Joel, can you please CR this?

#5 Updated by Miguel Angel Medinilla Luque almost 2 years ago

Augustus (http://augustus.qa.archivematica.org) was redeployed with the 'dev/issue-11213-dip-upload-fix' branch.

I have tested some DIP uploads with the atom-24rc3-test server (http://atom-24rc3-test.accesstomemory.org/).

#6 Updated by Mike Cantelon almost 2 years ago

Ok, the UI error is fixed in the new deployment. I ran into another issue that I was running into previously IIRC during the rsync part of the DIP upload and managed to find a workaround for.

When I manually replicated, to diagnose, the rsync command done during DIP upload I ended up getting a "code 23" error (incomplete). Using the -n flag, I was able to see what wasn't being transferred.

artefactual@augustus:~$ sudo -u archivematica rsync -n -e ssh -l archivematica 158.69.72.17 -i /var/lib/archivematica/.ssh/id_rsa --protect-args -rltz -P --chmod=ugo=rwX /var/archivematica/sharedDirectory/watchedDirectories/uploadDIP/Test4-e1f83cf2-03c3-43fe-8c5c-aa10458f6075 158.69.72.17:/tmp
sending incremental file list
rsync: link_stat "/home/artefactual/archivematica" failed: No such file or directory (2)
rsync: link_stat "/home/artefactual/158.69.72.17" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]

When I did

mkdir /home/artefactual/archivematica && /home/artefactual/158.69.72.17
the rsync would subsequently work.

#7 Updated by Mike Cantelon almost 2 years ago

It seems like now the Archmivematica side seems to be working, but for some reason I don't see the corresponding digital objects on the AtoM side so I'll see if I can figure out what the issue is.

#8 Updated by Mike Cantelon almost 2 years ago

Ah, getting a weird error on the AtoM side. I'll investigate.

2017-11-16 13:21:20 > Running worker...
2017-11-16 13:21:20 > PID 4814
2017-11-16 13:29:48 > Job started.
2017-11-16 13:29:48 > A package was deposited by reference.
2017-11-16 13:29:48 > Location: file:///MikeTest3-4bafc3e7-c3f3-4566-beeb-5f1d1031c87e
2017-11-16 13:29:48 > Processing...
2017-11-16 13:29:48 > Object slug: mike-test
2017-11-16 13:29:48 > Exception: Level of description not found: Directory

#9 Updated by Miguel Angel Medinilla Luque almost 2 years ago

I have deployed this morning a new clean AtoM and I get the same error at AtoM job report when testing DIP upload from augustus:

[info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": Job started.
[info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": A package was deposited by reference.
[info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": Location: file:///miguel_test6-310bd043-f466-4c38-b078-7a69610eddd6
[info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": Processing...
[info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": Object slug: memo
[info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": Exception: Level of description not found: Directory

#10 Updated by Kelly Stewart almost 2 years ago

I can confirm - DIP upload failing at Job: Choose config for AtoM DIP upload

#11 Updated by Nick Wilkinson almost 2 years ago

  • Status changed from Code Review to In progress
  • Assignee changed from Joel Dunham to Mike Cantelon

#12 Updated by Mike Cantelon almost 2 years ago

  • Related to Bug #11721: DIP upload logic incompatible with METS logical structMap added

#13 Updated by Mike Cantelon almost 2 years ago

Current status:

  • I've added a commit to AtoM qa/2.5.x to fix the underlying issue (the normative logical structmap being misinterpreted as a hierarchal logical structmap)
  • If we want to have AtoM DIP upload to AtoM 2.4.x. work with the upcoming version of Archivematica, however, we'll either have to do another point release of AtoM 2.4 or offer a patch
  • The Archivematica-side "Choose config for AtoM DIP upload" failure still needs to be fixed (it's not an actual failure, but a bug that a failure is reported)

I'm currently working on the Archivematica chain issue.

#14 Updated by Mike Cantelon almost 2 years ago

  • Status changed from In progress to Code Review
  • Assignee changed from Mike Cantelon to Nick Wilkinson

#15 Updated by Justin Simpson almost 2 years ago

  • Assignee changed from Nick Wilkinson to Justin Simpson

#16 Updated by Justin Simpson almost 2 years ago

  • Status changed from Code Review to QA/Review
  • Assignee changed from Justin Simpson to Sarah Romkey

miguel is deploying this now. It has been merged into stable/1.7.x and qa/1.x should be ready to test.

#17 Updated by Sarah Romkey almost 2 years ago

  • Status changed from QA/Review to Feedback
  • Assignee changed from Sarah Romkey to Mike Cantelon

A couple pieces of feedback:

1. This was tested with http://atom-24rc3-test.accesstomemory.org - it still fails to upload the DIP but I think that's expected? It's no longer failing silently, which is good. Here is the stderr:

upload-qubit.py: ERROR 2017-12-04 22:53:28,029 archivematica.upload.qubit:log:64: [uploadDIP] Target: sarah
upload-qubit.py: ERROR 2017-12-04 22:53:28,029 archivematica.upload.qubit:log:64: [uploadDIP] rsync -e None --protect-args -rltz -P --chmod=ugo=rwX /var/archivematica/sharedDirectory/watchedDirectories/uploadDIP/dip_test2-e1586b71-1a0a-47a5-b736-981f698114c7 158.69.72.17:/tmp
upload-qubit.py: ERROR 2017-12-04 22:53:28,033 archivematica.upload.qubit:log:64: [uploadDIP] Rsync output is being saved in /tmp/tmpouCojA
[uploadDIP] Rsync quit unexpectedly (exit 14), the upload script will be stopped here

2. After entering the AtoM slug, there is a new job with a choice for Choose config for AtoM DIP upload. The only choice in the dropdown is upload-qubit_V0.0. This is similar behaviour to the ArchivesSpace DIP upload, but it's not desirable unless it's related to a new feature (as discussed in Slack, we don't believe it is).

#18 Updated by Mike Cantelon almost 2 years ago

  • Related to Feature #11792: Give the user a choice, during AIP creation, whether or not they want a normative structMap in the METS XML added

#19 Updated by Mike Cantelon almost 2 years ago

Update: I've made a separate issue (#11792) for making METS structMap generation a user choice and submitted a PR adding this functionality.

#20 Updated by Mike Cantelon almost 2 years ago

  • Status changed from Feedback to QA/Review
  • Assignee changed from Mike Cantelon to Nick Wilkinson

I've merged the DIP fixes to AM stable/1.7.x and qa/1.x so if we deploy to the test server we can do QA on it.

#21 Updated by Nick Wilkinson almost 2 years ago

  • Status changed from QA/Review to Deploy
  • Assignee changed from Nick Wilkinson to Darren Craze

Hi Darren, this fix, along with #11792, can be deployed for testing on either am17qa or augustus, then passed to Kelly or Sarah for QA.

#22 Updated by Darren Craze almost 2 years ago

  • Assignee changed from Darren Craze to Sarah Romkey

Let us know which test environment you need updated to test this.

#23 Updated by Sarah Romkey almost 2 years ago

Santi had already updated Augustus but the failure is still occurring. We'll consult with a dev and get back to you.

#24 Updated by Mike Cantelon almost 2 years ago

I was about to upload a DIP to http://atom-24rc3-test.accesstomemory.org/mike-test successfully (selecting either "yes" or "no" for document empty directories).

#25 Updated by Sarah Romkey almost 2 years ago

  • Status changed from Deploy to Feedback
  • Assignee changed from Sarah Romkey to Mike Cantelon

It works, but is still prompting for the config. This will break some of our users' automated workflows. See comment #17 above.

#26 Updated by Mike Cantelon almost 2 years ago

I've created a PR that fixes this, but there's a cosmetic issue with it (the status in the dashboard for the selected configuration ends up displayed as "Unknown" rather than "Completed successfully") so I'm going to ask others to see if they have an idea how to fix.

https://github.com/artefactual/archivematica/pull/886

#27 Updated by Mike Cantelon over 1 year ago

  • Status changed from Feedback to Code Review
  • Assignee changed from Mike Cantelon to Nick Wilkinson

I've fixed the cosmetic issue in the above PR so it's ready for code review.

#28 Updated by Nick Wilkinson over 1 year ago

  • Assignee changed from Nick Wilkinson to Joel Dunham

Hi Joel, can you take a look for Code Review?

#29 Updated by Joel Dunham over 1 year ago

  • Assignee changed from Joel Dunham to Sarah Romkey

Hi Sarah,

I've approved this. Once Mike merges it, can you QA it?

#30 Updated by Mike Cantelon over 1 year ago

  • Status changed from Code Review to QA/Review
  • Assignee changed from Sarah Romkey to Nick Wilkinson

Merged into stable/1.7.x. I guess we should probably re-deploy the test server to QA it.

#31 Updated by Nick Wilkinson over 1 year ago

  • Assignee changed from Nick Wilkinson to Sarah Romkey

Hi Sarah, can you let Darren know where/when you need this deployed so it can be tested?

#32 Updated by Sarah Romkey over 1 year ago

We're asking the sys admins for a refresh of our test sites soon, I can test it there.

#33 Updated by Dan Gillean about 1 month ago

  • Category set to Access
  • Status changed from QA/Review to Verified

Also available in: Atom PDF