Bug #11419

Multi-process bagit.validate breaks AIP re-ingest

Added by Joel Dunham about 2 years ago. Updated about 2 years ago.

Status:VerifiedStart date:08/11/2017
Priority:HighDue date:
Assignee:Kelly Stewart% Done:

0%

Category:Reingest
Target version:Release 1.7.0
Google Code Legacy ID: Pull Request:https://github.com/artefactual/archivematica-storage-service/pull/230
Sponsored:No Requires documentation:

Description

For all the details, see GitHub issue https://github.com/artefactual/archivematica/issues/708.

To re-create: using AM qa/1.x and SS qa/0.x, create an AIP and then attempt to do a metadata-only reingest on it. AM should spin its wheels for 5 seconds and then notify you that "Error re-ingesting package: An unknown error occurred." If you insert some logging lines before and after https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/storage_service/locations/models/package.py#L843, you will see that the log messages after that line do not get called: the call to bag.validate(processes=4) hangs indefinitely. If you replace that call with the original bag.validate() everything works as normal.

Cause: Thanks to Jesús for figuring out the source of this: it's an incompatibility between Gunicorn's gevent worker class and BagIt's use of multiprocessing. In order for bag.validate(processes=4) to work, Gunicorn must be using the sync worker class.

History

#1 Updated by Joel Dunham about 2 years ago

  • Assignee changed from Joel Dunham to Nick Wilkinson
  • Pull Request set to https://github.com/artefactual/archivematica-storage-service/pull/230

Should be fixed by https://github.com/artefactual/archivematica-storage-service/pull/230 which is awaiting code review.

#2 Updated by Nick Wilkinson about 2 years ago

  • Assignee changed from Nick Wilkinson to Jesús García Crespo

Hi Jesús, assigning to you for CR.

#3 Updated by Joel Dunham about 2 years ago

  • Status changed from New to QA/Review
  • Assignee changed from Jesús García Crespo to Nick Wilkinson

#4 Updated by Nick Wilkinson about 2 years ago

  • Assignee deleted (Nick Wilkinson)

#5 Updated by Kelly Stewart about 2 years ago

  • Assignee set to Nick Wilkinson

Just tested this on augustus.qa (test transfer name AIPreingest) and the bug persists with same error message: Error re-ingesting package: An unknown error occurred.

https://github.com/artefactual/archivematica/issues/708

#6 Updated by Nick Wilkinson about 2 years ago

  • Status changed from QA/Review to Feedback
  • Assignee changed from Nick Wilkinson to Joel Dunham

Hi Joel, can you please take a look at this again? Kelly says the bug persists with same error message.

#7 Updated by Joel Dunham about 2 years ago

  • Assignee changed from Joel Dunham to Nick Wilkinson

This is a distinct issue, unrelated to bagit.validate. See RM https://projects.artefactual.com/issues/11519 and GH https://github.com/artefactual/archivematica-storage-service/issues/234 for further work on this bug.

#8 Updated by Nick Wilkinson about 2 years ago

  • Status changed from Feedback to QA/Review
  • Assignee changed from Nick Wilkinson to Kelly Stewart

Hi Kelly, considering the issue you saw is a separate issue (#11519) are you able to Verify this issue?

#9 Updated by Kelly Stewart about 2 years ago

  • Status changed from QA/Review to Verified

Also available in: Atom PDF