atom dip upload not working
|Assignee:||Sarah Romkey||% Done:|
|Target version:||Release 1.7.0|
|Google Code Legacy ID:||Pull Request:|
As described in https://github.com/artefactual/archivematica/issues/622
The atom dip upload feature is not working in qa/1.x
- When using the web interface, the question "Upload DIP" ->"Upload DIP to Atom" asks for the atom slug, but never finishes and the question is always in "Awaiting decision" state.
- Using am mcp-rpc-cli, when selecting "Upload DIP to Atom" for the "Upload DIP" question, there is a new question, with only one choice ( upload-qubit_v0.0):
<choicesAvailableForUnit> <UUID>3f857f46-6a10-472d-a564-77bd7534f58b</UUID> <unit> <type>DIP</type> <unitXML> <UUID>c8529576-a069-48fc-b056-8e260275e86c</UUID> <currentPath>%sharedPath%watchedDirectories/uploadDIP/atom4-c8529576-a069-48fc-b056-8e260275e86c/</currentPath> </unitXML> </unit> <choices> <choice> <chainAvailable>0</chainAvailable> <description>upload-qubit_v0.0</description> </choice> </choices> </choicesAvailableForUnit>
- The switch --rsync-command="None"\ is being passed to the script, and that makes rsync fail with error 14
#2 Updated by Nick Wilkinson over 2 years ago
- Assignee changed from Nick Wilkinson to Mike Cantelon
- Priority changed from High to Critical
Hi Mike, assigning to you to investigate. This should be considered a top priority. Please work closely with Kelly on this for more background on the bug and where you can deploy this so she can test again.
#3 Updated by Mike Cantelon over 2 years ago
- Status changed from New to Code Review
- Assignee changed from Mike Cantelon to Nick Wilkinson
There's still an issue with DIP upload that I'm trying to fix, but it's a separate issue and, I'd guess, might get resolved simply by re-deploying.
#6 Updated by Mike Cantelon over 2 years ago
Ok, the UI error is fixed in the new deployment. I ran into another issue that I was running into previously IIRC during the rsync part of the DIP upload and managed to find a workaround for.
When I manually replicated, to diagnose, the rsync command done during DIP upload I ended up getting a "code 23" error (incomplete). Using the
-n flag, I was able to see what wasn't being transferred.
artefactual@augustus:~$ sudo -u archivematica rsync -n -e ssh -l archivematica 126.96.36.199 -i /var/lib/archivematica/.ssh/id_rsa --protect-args -rltz -P --chmod=ugo=rwX /var/archivematica/sharedDirectory/watchedDirectories/uploadDIP/Test4-e1f83cf2-03c3-43fe-8c5c-aa10458f6075 188.8.131.52:/tmp sending incremental file list rsync: link_stat "/home/artefactual/archivematica" failed: No such file or directory (2) rsync: link_stat "/home/artefactual/184.108.40.206" failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]
When I did
mkdir /home/artefactual/archivematica && /home/artefactual/220.127.116.11the rsync would subsequently work.
#8 Updated by Mike Cantelon over 2 years ago
Ah, getting a weird error on the AtoM side. I'll investigate.
2017-11-16 13:21:20 > Running worker... 2017-11-16 13:21:20 > PID 4814 2017-11-16 13:29:48 > Job started. 2017-11-16 13:29:48 > A package was deposited by reference. 2017-11-16 13:29:48 > Location: file:///MikeTest3-4bafc3e7-c3f3-4566-beeb-5f1d1031c87e 2017-11-16 13:29:48 > Processing... 2017-11-16 13:29:48 > Object slug: mike-test 2017-11-16 13:29:48 > Exception: Level of description not found: Directory
#9 Updated by Miguel Angel Medinilla Luque over 2 years ago
I have deployed this morning a new clean AtoM and I get the same error at AtoM job report when testing DIP upload from augustus:
[info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": Job started. [info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": A package was deposited by reference. [info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": Location: file:///miguel_test6-310bd043-f466-4c38-b078-7a69610eddd6 [info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": Processing... [info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": Object slug: memo [info] [2017-11-17 03:19:36] Job 2001305 "qtSwordPluginWorker": Exception: Level of description not found: Directory
#13 Updated by Mike Cantelon over 2 years ago
- I've added a commit to AtoM qa/2.5.x to fix the underlying issue (the normative logical structmap being misinterpreted as a hierarchal logical structmap)
- If we want to have AtoM DIP upload to AtoM 2.4.x. work with the upcoming version of Archivematica, however, we'll either have to do another point release of AtoM 2.4 or offer a patch
- The Archivematica-side "Choose config for AtoM DIP upload" failure still needs to be fixed (it's not an actual failure, but a bug that a failure is reported)
I'm currently working on the Archivematica chain issue.
#14 Updated by Mike Cantelon over 2 years ago
- Status changed from In progress to Code Review
- Assignee changed from Mike Cantelon to Nick Wilkinson
#17 Updated by Sarah Romkey over 2 years ago
- Status changed from QA/Review to Feedback
- Assignee changed from Sarah Romkey to Mike Cantelon
A couple pieces of feedback:
1. This was tested with http://atom-24rc3-test.accesstomemory.org - it still fails to upload the DIP but I think that's expected? It's no longer failing silently, which is good. Here is the stderr:
upload-qubit.py: ERROR 2017-12-04 22:53:28,029 archivematica.upload.qubit:log:64: [uploadDIP] Target: sarah
upload-qubit.py: ERROR 2017-12-04 22:53:28,029 archivematica.upload.qubit:log:64: [uploadDIP] rsync -e None --protect-args -rltz -P --chmod=ugo=rwX /var/archivematica/sharedDirectory/watchedDirectories/uploadDIP/dip_test2-e1586b71-1a0a-47a5-b736-981f698114c7 18.104.22.168:/tmp
upload-qubit.py: ERROR 2017-12-04 22:53:28,033 archivematica.upload.qubit:log:64: [uploadDIP] Rsync output is being saved in /tmp/tmpouCojA
[uploadDIP] Rsync quit unexpectedly (exit 14), the upload script will be stopped here
2. After entering the AtoM slug, there is a new job with a choice for Choose config for AtoM DIP upload. The only choice in the dropdown is upload-qubit_V0.0. This is similar behaviour to the ArchivesSpace DIP upload, but it's not desirable unless it's related to a new feature (as discussed in Slack, we don't believe it is).
#24 Updated by Mike Cantelon over 2 years ago
I was about to upload a DIP to http://atom-24rc3-test.accesstomemory.org/mike-test successfully (selecting either "yes" or "no" for document empty directories).
#26 Updated by Mike Cantelon over 2 years ago
I've created a PR that fixes this, but there's a cosmetic issue with it (the status in the dashboard for the selected configuration ends up displayed as "Unknown" rather than "Completed successfully") so I'm going to ask others to see if they have an idea how to fix.