Feature #13573

Atom worker service - add 'max-job-count' and 'max-mem-limit' options

Added by Steve Breker about 1 month ago. Updated about 1 month ago.

Status:QA/ReviewStart date:10/05/2021
Priority:HighDue date:
Assignee:Steve Breker% Done:

0%

Category:Job scheduling
Target version:Release 2.7.0
Google Code Legacy ID: Tested version:
Sponsored:No Requires documentation:Yes

Description

Add a new option to the jobs:worker task to accept a max-job-count value. This max-job-count value will indicate to the atom-worker the number of jobs it should process before shutting down. The atom-worker will return a specific value (return code 111) when it shuts down due to the maximum number of jobs being reached. This return can be checked by systemd to trigger a restart of the atom-worker.

This option can be used when configuring the atom-worker service (/usr/lib/systemd/system/atom-worker.service).

e.g. Set to shutdown after 10 jobs completed:
ExecStart=/usr/bin/php7.4 -d memory_limit=1000M -d error_reporting="E_ALL" -dextension=memprof.so symfony jobs:worker --max-job-count=10

Similarly, add a new --max-mem-usage option that will cause the worker to exit with code '111' when the atom-worker hits this configured limit. Jobs must not be stopped midway, but must be allowed to complete after the mem limit is breached.

e.g. Set to shutdown after 10 jobs completed OR if the atom-worker res memory exceeds 500MB:
ExecStart=/usr/bin/php7.4 -d memory_limit=1000M -d error_reporting="E_ALL" -dextension=memprof.so symfony jobs:worker --max-job-count=10 --max-mem-limit=500000

https://www.accesstomemory.org/en/docs/2.7/admin-manual/installation/ubuntu/#deployment-of-workers


Related issues

Related to Access to Memory (AtoM) - Bug #13109: AtoM worker service is unstable in systemd Verified 07/12/2019
Related to Access to Memory (AtoM) - Feature #13575: AtoM worker memory usage In progress 10/12/2021

History

#1 Updated by Steve Breker about 1 month ago

  • Subject changed from atom-worker --max-job-count to Atom worker service - add max-job-count option

#2 Updated by Steve Breker about 1 month ago

  • Description updated (diff)

#4 Updated by Steve Breker about 1 month ago

  • Description updated (diff)

#5 Updated by Steve Breker about 1 month ago

  • Related to Bug #13109: AtoM worker service is unstable in systemd added

#6 Updated by Steve Breker about 1 month ago

  • Requires documentation set to Yes

#8 Updated by Steve Breker about 1 month ago

#9 Updated by Steve Breker about 1 month ago

PR for --max-mem-usage option:

https://github.com/artefactual/atom/pull/1453

#10 Updated by Steve Breker about 1 month ago

  • Description updated (diff)

#11 Updated by Steve Breker about 1 month ago

The max-mem-usage option will not shutdown immediately. This limit is checked between jobs so that they have a chance to complete before the worker shuts down with exit code 111. This means that --max-mem-usage must be set LOWER (25-50% lower?) than the PHP memory limit to account for large jobs completing.

#12 Updated by Steve Breker about 1 month ago

  • Subject changed from Atom worker service - add max-job-count option to Atom worker service - add 'max-job-count' and 'max-mem-limit' options

#13 Updated by Steve Breker about 1 month ago

  • Status changed from New to QA/Review

Also available in: Atom PDF