Task #10560

Add DSpace space to the storage service

Added by Sara Allain over 5 years ago. Updated over 5 years ago.

Status:QA/ReviewStart date:11/16/2016
Priority:MediumDue date:
Assignee:Sarah Romkey% Done:

0%

Category:-
Target version:0.10.0
Sponsored:No Requires documentation:

Description

This feature will add a DSpace Space to the storage service, using the following workflow:

  • POST AIP to DSpace using SWORD2 API
  • Parse AIP DublinCore metadata and send with SWORD2
  • Split package into two bitstreams: objects and metadata
  • Set custom permissions on the metadata bitstream
  • Return misc attributes on a Package in metadata
  • move_from_storage_service accepts a Package parameter

History

#3 Updated by Sarah Romkey over 5 years ago

  • Status changed from QA/Review to Feedback
  • Assignee changed from Sarah Romkey to Nick Wilkinson

I have created a space and an AIP storage location with the exact credentials on the missy qa server. Could you please have a dev take a look? I am getting this error at Store AIP:

Command: storeAIP_v0.0 "/api/v2/location/40d5fe67-f483-4fe9-8abb-144ffcbafb31/" "/var/archivematica/sharedDirectory/currentlyProcessing/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z" "011b27f5-ef07-4c8e-89b5-b870127c1df9" "dspace_again" "SIP"

STDERR

SIP creation failed. See Storage Service logs for more details
Server Error 500: http://198.168.168.198:8000/api/v2/file/

SS debug log:

DEBUG 2016-12-13 15:41:36 common.utils:utils:uuid_to_path:249: path 011b/27f5/ef07/4c8e/89b5/b870/127c/1df9
DEBUG 2016-12-13 15:41:36 locations.models.space:space:move_to_storage_service:264: TO: src: var/archivematica/sharedDirectory/currentlyProcessing/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9/pointer.xml
DEBUG 2016-12-13 15:41:36 locations.models.space:space:move_to_storage_service:265: TO: dst: 011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/pointer.011b27f5-ef07-4c8e-89b5-b870127c1df9.xml
DEBUG 2016-12-13 15:41:36 locations.models.space:space:move_to_storage_service:266: TO: staging: /var/archivematica/storage_service
INFO 2016-12-13 15:41:36 locations.models.space:space:move_rsync:421: Moving from /var/archivematica/sharedDirectory/currentlyProcessing/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9/pointer.xml to /var/archivematica/sto
rage_service/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/pointer.011b27f5-ef07-4c8e-89b5-b870127c1df9.xml
INFO 2016-12-13 15:41:36 locations.models.space:space:move_rsync:449: rsync command: ['rsync', '-t', '-O', '--protect-args', '-vv', '--chmod=ugo+rw', '-r', '/var/archivematica/sharedDirectory/currentlyProcessing/dspace_again
-011b27f5-ef07-4c8e-89b5-b870127c1df9/pointer.xml', '/var/archivematica/storage_service/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/pointer.011b27f5-ef07-4c8e-89b5-b870127c1df9.xml']
DEBUG 2016-12-13 15:41:36 locations.models.space:space:move_from_storage_service:333: FROM: src: 011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/pointer.011b27f5-ef07-4c8e-89b5-b870127c1df9.xml
DEBUG 2016-12-13 15:41:36 locations.models.space:space:move_from_storage_service:334: FROM: dst: var/archivematica/storage_service/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/pointer.011b27f5-ef07-4c8e-89b5-b870127c1df9.xml
INFO 2016-12-13 15:41:36 locations.models.space:space:move_rsync:421: Moving from /var/archivematica/storage_service/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/pointer.011b27f5-ef07-4c8e-89b5-b870127c1df9.xml to /var/archivemat
ica/storage_service/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/pointer.011b27f5-ef07-4c8e-89b5-b870127c1df9.xml
DEBUG 2016-12-13 15:41:36 locations.models.space:space:move_to_storage_service:264: TO: src: var/archivematica/sharedDirectory/currentlyProcessing/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9/dspace_again-011b27f5-ef07-4
c8e-89b5-b870127c1df9.7z
DEBUG 2016-12-13 15:41:36 locations.models.space:space:move_to_storage_service:265: TO: dst: 011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z
DEBUG 2016-12-13 15:41:36 locations.models.space:space:move_to_storage_service:266: TO: staging: /var/archivematica/storage_service/
INFO 2016-12-13 15:41:36 locations.models.space:space:move_rsync:421: Moving from /var/archivematica/sharedDirectory/currentlyProcessing/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z to /var/archivematica/storage_service/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z
INFO 2016-12-13 15:41:36 locations.models.space:space:move_rsync:449: rsync command: ['rsync', '-t', '-O', '--protect-args', '-vv', '--chmod=ugo+rw', '-r', '/var/archivematica/sharedDirectory/currentlyProcessing/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z', '/var/archivematica/storage_service/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z']
DEBUG 2016-12-13 15:41:36 locations.models.space:space:move_from_storage_service:333: FROM: src: 011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z
DEBUG 2016-12-13 15:41:36 locations.models.space:space:move_from_storage_service:334: FROM: dst: http://dspace5x.archivematica.org:8080/swordv2/collection/123456789/2/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z
INFO 2016-12-13 15:41:36 locations.models.dspace:dspace:move_from_storage_service:197: source_path: /var/archivematica/storage_service/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z, destination_path: http://dspace5x.archivematica.org:8080/swordv2/collection/123456789/2/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z, package: 011b27f5-ef07-4c8e-89b5-b870127c1df9: http:/dspace5x.archivematica.org:8080/swordv2/collection/123456789/2/011b/27f5/ef07/4c8e/89b5/b870/127c/1df9/dspace_again-011b27f5-ef07-4c8e-89b5-b870127c1df9.7z
DEBUG 2016-12-13 15:41:36 locations.models.dspace:dspace:_get_sword_connection:60: Getting sword connection
INFO 2016-12-13 15:41:36 sword2.connection:connection:__init__:217: Loading default HTTP layer
ERROR 2016-12-13 15:41:36 django.request.tastypie:resources:_handle_500:301: Internal Server Error: /api/v2/file/
Traceback (most recent call last):
File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/tastypie/resources.py", line 220, in wrapper
response = callback(request, *args, **kwargs)
File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/tastypie/resources.py", line 451, in dispatch_list
return self.dispatch('list', request, **kwargs)
File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/tastypie/resources.py", line 483, in dispatch
response = method(request, **kwargs)
File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/tastypie/resources.py", line 1380, in post_list
updated_bundle = self.obj_create(bundle, **self.remove_api_resource_names(kwargs))
File "./locations/api/resources.py", line 472, in obj_create
bundle.obj.store_aip(origin_location, origin_path, related_package_uuid)
File "./locations/models/package.py", line 461, in store_aip
package=self,
File "./locations/models/space.py", line 340, in move_from_storage_service
source_path, destination_path, *args, **kwargs)
File "./locations/models/dspace.py", line 206, in move_from_storage_service
self._get_sword_connection()
File "./locations/models/dspace.py", line 66, in get_sword_connection
keep_history=True,
File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/sword2/connection.py", line 218, in init
self.h = http_layer.HttpLib2Layer(".cache", timeout=30.0)
File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/sword2/http_layer.py", line 58, in init
self.h = httplib2.Http(".cache", timeout=30.0)
File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/httplib2/
_init__.py", line 1192, in init
self.cache = FileCache(cache)
File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/httplib2/__init__.py", line 700, in init
os.makedirs(self.cache)
File "/usr/share/python/archivematica-storage-service/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '.cache'

#4 Updated by Nick Wilkinson over 5 years ago

  • Assignee changed from Nick Wilkinson to Holly Becker

#5 Updated by Sarah Romkey over 5 years ago

In addition to the errors, I note that the AIPs I attempted to send to DSpace are listed in the Storage Service packages tab as "Staged on Storage Service".

#6 Updated by Holly Becker over 5 years ago

  • Status changed from Feedback to Code Review
  • Assignee changed from Holly Becker to Nick Wilkinson

It was failing because it didn't have permission to create the `.cache` directory the sword2 library creates by default. Since this has caused problems several times and provides little benefit, updated to remove the .cache directory entirely. This includes an update to the sword2 library to enable this change.

The package status of the failed packages is correct - since the transfer to the final storage place didn't work, the best status they achieved was staged to the storage service.

Ready for code review & testing before merge.

#7 Updated by Nick Wilkinson over 5 years ago

  • Assignee changed from Nick Wilkinson to Joel Dunham

#8 Updated by Joel Dunham over 5 years ago

  • Status changed from Code Review to Feedback
  • Assignee changed from Joel Dunham to Nick Wilkinson

I've CR-ed and approved SS PR #169.

#9 Updated by Nick Wilkinson over 5 years ago

  • Assignee changed from Nick Wilkinson to Holly Becker

#10 Updated by Holly Becker over 5 years ago

  • Status changed from Feedback to QA/Review
  • Assignee changed from Holly Becker to Sarah Romkey

Merged into qa/0.x.

#11 Updated by Sarah Romkey over 5 years ago

  • Status changed from QA/Review to Feedback
  • Assignee changed from Sarah Romkey to Holly Becker

AIPs can now be stored in DSpace but I receive the following error when trying to download from Archival Storage:

<response><error_message>Sorry, this request could not be processed. Please try again later.</error_message></response>

Holly, could you please take a look?

#12 Updated by Holly Becker over 5 years ago

  • Assignee changed from Holly Becker to Sarah Romkey

Correct - fetching AIPs from DSpace using the Storage Service was not implemented. I think download from DSpace itself was considered sufficient for the initial implementation.

Internally it raises a NotImplemented exception, but that error message doesn't appear when not in debug mode. It might be worthwhile to display that in production to reduce confusion.

#13 Updated by Sarah Romkey over 5 years ago

  • Status changed from Feedback to QA/Review

Also available in: Atom PDF