Bug #11419
Multi-process bagit.validate breaks AIP re-ingest
Status: | Verified | Start date: | 08/11/2017 | |
---|---|---|---|---|
Priority: | High | Due date: | ||
Assignee: | Kelly Stewart | % Done: | 0% | |
Category: | Reingest | |||
Target version: | Release 1.7.0 | |||
Google Code Legacy ID: | Pull Request: | https://github.com/artefactual/archivematica-storage-service/pull/230 | ||
Sponsored: | No | Requires documentation: |
Description
For all the details, see GitHub issue https://github.com/artefactual/archivematica/issues/708.
To re-create: using AM qa/1.x and SS qa/0.x, create an AIP and then attempt to do a metadata-only reingest on it. AM should spin its wheels for 5 seconds and then notify you that "Error re-ingesting package: An unknown error occurred." If you insert some logging lines before and after https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/storage_service/locations/models/package.py#L843, you will see that the log messages after that line do not get called: the call to bag.validate(processes=4) hangs indefinitely. If you replace that call with the original bag.validate() everything works as normal.
Cause: Thanks to Jesús for figuring out the source of this: it's an incompatibility between Gunicorn's gevent worker class and BagIt's use of multiprocessing. In order for bag.validate(processes=4) to work, Gunicorn must be using the sync worker class.
History
#1 Updated by Joel Dunham almost 5 years ago
- Assignee changed from Joel Dunham to Nick Wilkinson
- Pull Request set to https://github.com/artefactual/archivematica-storage-service/pull/230
Should be fixed by https://github.com/artefactual/archivematica-storage-service/pull/230 which is awaiting code review.
#2 Updated by Nick Wilkinson almost 5 years ago
- Assignee changed from Nick Wilkinson to Jesús García Crespo
Hi Jesús, assigning to you for CR.
#3 Updated by Joel Dunham almost 5 years ago
- Status changed from New to QA/Review
- Assignee changed from Jesús García Crespo to Nick Wilkinson
#4 Updated by Nick Wilkinson almost 5 years ago
- Assignee deleted (
Nick Wilkinson)
#5 Updated by Kelly Stewart over 4 years ago
- Assignee set to Nick Wilkinson
Just tested this on augustus.qa (test transfer name AIPreingest) and the bug persists with same error message: Error re-ingesting package: An unknown error occurred.
#6 Updated by Nick Wilkinson over 4 years ago
- Status changed from QA/Review to Feedback
- Assignee changed from Nick Wilkinson to Joel Dunham
Hi Joel, can you please take a look at this again? Kelly says the bug persists with same error message.
#7 Updated by Joel Dunham over 4 years ago
- Assignee changed from Joel Dunham to Nick Wilkinson
This is a distinct issue, unrelated to bagit.validate. See RM https://projects.artefactual.com/issues/11519 and GH https://github.com/artefactual/archivematica-storage-service/issues/234 for further work on this bug.
#8 Updated by Nick Wilkinson over 4 years ago
- Status changed from Feedback to QA/Review
- Assignee changed from Nick Wilkinson to Kelly Stewart
Hi Kelly, considering the issue you saw is a separate issue (#11519) are you able to Verify this issue?
#9 Updated by Kelly Stewart over 4 years ago
- Status changed from QA/Review to Verified