Bug #13085

Updating an authority record or repository can cause the atom-worker to crash

Added by David Juhasz 4 months ago. Updated 4 months ago.

Status:VerifiedStart date:06/12/2019
Priority:MediumDue date:
Assignee:-% Done:

0%

Category:Job scheduling
Target version:Release 2.5.1
Google Code Legacy ID: Tested version:2.5, 2.6
Sponsored:No Requires documentation:

Description

To reproduce
  1. Login to AtoM with permissions to edit an authority record
  2. Find an authority record linked to a large number of descriptions as a creator or subject, or an archival institution holding many descriptions (I've reproduced this error with a repository holding ~500 descriptions)
  3. Change the name of the authority record or archival institution and click "Save"

Error encountered
The AtoM worker crashes with an error message when the job output exceeds the size allowed by the jobs table:

SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'output' at row 1

Expected result
The search index document for all archival descriptions related to the authority record should be updated with the new authority record name.

Notes
- As part of the enhancements for issue #11855, updates to the authority record now trigger the job scheduler to update the related search index records asynchoronously
- The code changes for issue #11855 were added in commit 794abc
- The job scheduler concatenates all job output to a single string and tries to store it to the job.output column - in this case the output has exceeded the 64kb limit on the column

atom_issue_13085_test_data.zip (3.02 MB) David Juhasz, 06/14/2019 10:59 AM


Related issues

Related to Access to Memory (AtoM) - Bug #11855: Search index is not updated when edits affect descendant ... Verified 01/10/2018

History

#1 Updated by David Juhasz 4 months ago

  • Subject changed from Changing authority record name can cause atom worker to crash to Changing authority record name can cause the atom-worker to crash

#2 Updated by David Juhasz 4 months ago

  • Related to Bug #11855: Search index is not updated when edits affect descendant records added

#3 Updated by David Juhasz 4 months ago

  • Subject changed from Changing authority record name can cause the atom-worker to crash to Updating an authority record can cause the atom-worker to crash

#4 Updated by David Juhasz 4 months ago

  • Description updated (diff)

#5 Updated by David Juhasz 4 months ago

  • File atom_issue_13085_test_data.zip added

#6 Updated by David Juhasz 4 months ago

I've modified our standard sample data to allow reproducing this error, by makeing all of the descriptions part to the "Artefactual Archives" holdings (file attached). To reproduce the error with the sample data:

  1. Download and unzip the sample data file
  2. Load the sample data to your AtoM instance, e.g. mysql -u root -p atom < atom_issue_13085_test_data.sql
  3. Populate the search index ./symfony search:populate
  4. Login to your AtoM site in your browser
  5. Go to the "Artefactual Archives" repository and change any data
  6. Click "save"
  7. The job manager page should show a running "arUpdateEsIoDocumentsJob" job with the info field stating Updating 476 description(s) and their descendants.
  8. Clicking the full report will show a detailed report of all descriptions updated
  9. After several minutes the job should crash the worker, you can confirm from the command line with sudo service atom-worker status which should show an "Active" value like Active: inactive (dead) since Thu 2019-06-13 22:59:58 UTC; 29min ago
To get the atom worker running again, you'll have to clear the job queue first:
  • ./symfony jobs:clear
  • sudo service atom-worker start

#7 Updated by David Juhasz 4 months ago

  • Subject changed from Updating an authority record can cause the atom-worker to crash to Updating an authority record or repository can cause the atom-worker to crash
  • Description updated (diff)

#8 Updated by Dan Gillean 4 months ago

It would be nice if we could reconfigure this job to run only when it actually affects fields displayed on the related records - i.e. generally the authorized form of name, which might appear in a hyperlink on related descriptions. Any other changes don't materially affect the related descriptions and as far as I can think of, shouldn't require an update of the descriptions.

#9 Updated by David Juhasz 4 months ago

Dan Gillean wrote:

It would be nice if we could reconfigure this job to run only when it actually affects fields displayed on the related records - i.e. generally the authorized form of name, which might appear in a hyperlink on related descriptions. Any other changes don't materially affect the related descriptions and as far as I can think of, shouldn't require an update of the descriptions.

I've been thinking the same thing Dan. It's more complicated than simply checking for a change to authorized form of name though; For creator/event relationships the admin history and maybe dates of existence are stored in the archival description search document. For repositories I'm not clear yet if any fields besides the authorized form of name are stored in the description docs. For "subject of" relationships I think only the authorized form of name is relevant.

#10 Updated by David Juhasz 4 months ago

  • File deleted (atom_issue_13085_test_data.zip)

#11 Updated by David Juhasz 4 months ago

Fixed name of "Artefactual Archives" in test data.

#12 Updated by David Juhasz 4 months ago

As a quick fix for AtoM 2.5.1 I've reduced the verbosity of the ES document update job in PR #913.

I have other concerns about the current job's scalability because I think it is updating far more ES documents than is required for many changes - for instance updating a creator's "Places" information or an archival institutions "Opening times" should not trigger the update of related archival description documents as these data are not relevant in the archival description context. These other scalability concerns will need to be addressed as part of qa/2.6.x or later though, as they will require significant code changes.

#13 Updated by David Juhasz 4 months ago

  • Status changed from In progress to Code Review
  • Assignee deleted (David Juhasz)

#14 Updated by Steve Breker 4 months ago

Looks great! Added one nitpicky request on there wrt the final output message.

#15 Updated by David Juhasz 4 months ago

  • Assignee set to Steve Breker

Thanks Steve. I've fixed the possibility of duplicating the final log message when document counts are a multiple of 100. Can you please double check the updated PR for me?

#16 Updated by David Juhasz 4 months ago

  • Status changed from Code Review to QA/Review
  • Assignee deleted (Steve Breker)

Merged to qa/2.6.x and ready for QA.

#17 Updated by Dan Gillean 4 months ago

Not sure what to do about this one. Tested multiple different times with my own dataset, and with David's attached to this issue - the job always ran successfully for me in stable.2.5.x, taking about 5 minutes to complete. I tested by upating the artefactual archives repository name.

I tried it again in qa/2.6.x and while the log is definitely less verbose, interestingly the task took 7 minutes to run.

#18 Updated by Corinne Rogers 4 months ago

  • Status changed from QA/Review to Verified

Tested changing the repository name for 1338 top level descriptions and descendants (total of 54,457 records) and all updated without a problem (https://hmm.accesstomemory.net/)

Also available in: Atom PDF