Bug #10187

Re-ingest failing on CentOS pipeline

Added by Sarah Romkey almost 6 years ago. Updated over 5 years ago.

Status:VerifiedStart date:08/05/2016
Priority:MediumDue date:
Assignee:Justin Simpson% Done:

0%

Category:Reingest
Target version:Release 1.5.1
Google Code Legacy ID: Pull Request:https://github.com/artefactual/archivematica-storage-service/pull/140
Sponsored:No Requires documentation:

Description

I tried reingesting an AIP on the churumelo pipeline and it failed with this error:

"Error re-ingesting package: An unknown error occurred."

Nick, could you please have a dev look into this?

History

#1 Updated by Nick Wilkinson almost 6 years ago

  • Assignee changed from Nick Wilkinson to Jesús García Crespo

Hi Jesús, can you look into this?

#2 Updated by Jesús García Crespo almost 6 years ago

  • Assignee changed from Jesús García Crespo to Santiago Collazo

Santi, I can't inspect SS logs because they are empty, I guess this is due to that unsolved issue where logging from Django in CentOS wasn't working. I remember that was a tricky issue that we tried to solve for a while without much luck. I'm assigning this ticket to you but I'm going to discuss with Nick, perhaps it's worth having a developer looking at it even though it has big chances to be an issue with the way the application's being packaged. Let me know.

On the dashboard side, I've found two different reingest attempts:

ERROR     2016-08-02 20:26:18  archivematica.common:storageService:request_reingest:374:  Unable to reingest 7d4a463f-3a9b-4dce-849e-e748af74078b
ERROR     2016-08-05 21:52:01  archivematica.common:storageService:request_reingest:374:  Unable to reingest 2da84892-43c9-4709-90d4-85b6f39fce3e

#3 Updated by Santiago Collazo almost 6 years ago

Ok , I can look about this again, and try to reach a solution.

#4 Updated by Santiago Collazo almost 6 years ago

  • Status changed from New to Feedback
  • Assignee changed from Santiago Collazo to Jesús García Crespo

There are some files from the "test_from_reingest" aip that fail due to a checksums problem:

ago 12 14:15:32 churumelo.qa.archivematica.org gunicorn[13948]: test_for_reingest-2da84892-43c9-4709-90d4-85b6f39fce3e/tagmanifest-md5.txt  (145 B)... OK.
ago 12 14:15:32 churumelo.qa.archivematica.org gunicorn[13948]: test_for_reingest-2da84892-43c9-4709-90d4-85b6f39fce3e/data/objects/funky_breakbeat_4.wav  (333486 B)... Failed! (Wrong checksum)
ago 12 14:15:32 churumelo.qa.archivematica.org gunicorn[13948]: test_for_reingest-2da84892-43c9-4709-90d4-85b6f39fce3e/data/objects/BigTeen_Short1-c26fe2aa-c90a-4f7c-92b1-daeb0901c2b4.wav  (7128744 B)... Failed! (Wrong checksum)
ago 12 14:15:32 churumelo.qa.archivematica.org gunicorn[13948]: test_for_reingest-2da84892-43c9-4709-90d4-85b6f39fce3e/data/objects/sample-c7f9e958-3539-46c9-baba-2aba21bb27f3.wav  (140850 B)... Failed! (Wrong checksum)
ago 12 14:15:32 churumelo.qa.archivematica.org gunicorn[13948]: Extraction to directory "/var/archivematica/storage_service/tmpAFSrUm" failed (3 files failed.)

Other AIPs work on both, churumelo and my local test environment, so I'm not sure if the problem is with this transfer, as if the files suffered some kind of corruption.

SS logs can be retrieved using journalctl, (journalctl -xn, journalctl | grep gunicorn ... )

#5 Updated by Jesús García Crespo almost 6 years ago

  • Assignee changed from Jesús García Crespo to Santiago Collazo

I am pretty sure that is unar.

#6 Updated by Santiago Collazo almost 6 years ago

Aside from the reingest error, storage-server own logs are not avaliable on journalctl, as you noted, only the output of external commands is loggued.

#7 Updated by Santiago Collazo almost 6 years ago

Removing the --access-log and --error log from the script, makes logs avaliable in journalctl (the output of the gunicorn process is managed by systemd)

#8 Updated by Santiago Collazo almost 6 years ago

  • Assignee changed from Santiago Collazo to Jesús García Crespo

I missjudged some lines in journalctl, that come from mcp-server/client, not from the storage-service, the current status is as follow:

- The storage-service is not writing any log on centos, doesn't matter if started as service, or "by hand", as archivematica user and after loading the environment variables needed.

- The problem also happens when deploying ss from source, and using virtualenv (instead of rpms). This deployment is in churumelo:/home/test/(venv|archivematica-storage-service).

Any idea?

#9 Updated by Jesús García Crespo almost 6 years ago

In CentOS, we install sword2 via pypi instead of using the wheel provided in the archiveamtica-storage-service repository.

I believe that we should update our call to `pip install` in our rpm spec with the extra --find-links argument pointing to the location of our wheels directory.


The old version of sword2 available @ pypi modifies the configuration of the logging module during the startup of the application, see sword2_logging.py for more details. In particular, it replaces the handlers that SS assigns the root handler from `settings/basic.py`.

The logging configuration injected by sword2 is pretty basic. The culprit is in:

[logger_root]
level=INFO
handlers=consoleHandler

I believe that the section above nulls our configuration for the root logger, which is:

    'root': {
        'handlers': ['logfile', 'verboselogfile'],
        'level': 'WARNING',
    },

Their full config is:

[loggers]
keys=root

[handlers]
keys=consoleHandler

[formatters]
keys=basicFormatting

[logger_root]
level=INFO
handlers=consoleHandler

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=basicFormatting
args=(sys.stdout,)

[formatter_basicFormatting]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s

#10 Updated by Jesús García Crespo almost 6 years ago

I've updated c#9 with my new findings. I've realized that we provide our own wheel for sword2 and our rpm spec was ignoring it. More details above.

#11 Updated by Jesús García Crespo almost 6 years ago

  • Assignee changed from Jesús García Crespo to Santiago Collazo

#12 Updated by Santiago Collazo over 5 years ago

  • Status changed from Feedback to Code Review
  • Assignee changed from Santiago Collazo to Jesús García Crespo

To avoid the use of "magic" files, I created https://github.com/artefactual/archivematica-storage-service/pull/140

This allows for installing storage-service in centos from source too, as is the same commit we had in ss-libs.

#13 Updated by Jesús García Crespo over 5 years ago

  • Assignee changed from Jesús García Crespo to Justin Simpson

#14 Updated by Jesús García Crespo over 5 years ago

  • Pull Request set to https://github.com/artefactual/archivematica-storage-service/pull/140

#15 Updated by Jesús García Crespo over 5 years ago

  • Status changed from Code Review to Verified

Also available in: Atom PDF