Feature #6087

Improve the upload calculation widget in Admin > Settings so it will not time out when checking large disks

Added by David Juhasz over 7 years ago. Updated almost 5 years ago.

Status:NewStart date:12/05/2013
Priority:MediumDue date:
Assignee:-% Done:

0%

Category:-Estimated time:20.00 hours
Target version:-
Sponsored:No Tested version:2.0.0, 2.0.1, 2.1, 2.1.1, 2.1.2, 2.2, 2.2.1, 2.3

Description

When an institution has a large number of digital objects in the uploads/ folder, then the "Total space available for uploads" widget in Admin > Settings can cause the page load to take a long time or timeout altogether while calculating the disk space used by the digital objects - see:

This widget was also used on archival description edit pages, adding further timeout issues when trying to edit existing descriptions.

We resolved the description edit page issues by removing the widget in issue #9275. Additionally, in issue #9674 we improved the behavior on the settings page by disabling the check when the widget is set to unlimited.

However, it would still be great to improve the widget so when it is set to a specific integer value, it does not cause timeouts, regardless of the size of the disk to be checked.

There are at least 2 possible approaches to improving this feature. One solution would be to calculate the disk space usage via an asynchronous task, using the job scheduler (introduced in issue #239). Another solution, instead of making the count update asynchronously (would we need a page reload to get the value?), might be:

  • Add a new DB column to one of the tables where this value can be stored - update it whenever a user uploads or deletes digital objects. This way we don't have to recalculate the value every single time.
  • There is a small risk that, if something is deleted from the filesystem, AtoM wouldn't know. We could potentially make a simple CLI task that would re-run the check and update the value in the DB, to make sure it remains consistent. In most cases this won't be an issue, but it would be a good idea to cover all eventualities.

Related issues

Related to Access to Memory (AtoM) - Bug #9275: Repository directory size check present when editing arch... Verified 01/06/2016
Related to Access to Memory (AtoM) - Feature #9674: Disable disk usage check when site upload limit is set to... Verified 04/07/2016
Related to Access to Memory (AtoM) - Feature #13411: Add setting to enable or disable repository "upload limit... Verified 08/27/2020
Duplicated by Access to Memory (AtoM) - Bug #8468: Digital object storage widget in Admin > Settings can tim... Duplicate 05/20/2015
Blocked by Access to Memory (AtoM) - Feature #239: Job scheduling support Verified 12/05/2011

History

#1 Updated by David Juhasz over 7 years ago

  • Description updated (diff)

#3 Updated by Dan Gillean about 6 years ago

  • Duplicated by Bug #8468: Digital object storage widget in Admin > Settings can time out settings page on sites with many digital objects added

#4 Updated by Mike Gale over 5 years ago

  • Related to Bug #9275: Repository directory size check present when editing archival descriptions added

#5 Updated by Dan Gillean over 5 years ago

See also issue #9275 - the solution there is to remove the widget, but since the same code is used, the timeouts could also affect the repository edit page (the other place where that same widget is used) if the repo has a lot of digital object uploads.

Another solution, instead of making the count update asynchronously (would we need a page reload to get the value?), might be:

  • Add a new DB column to one of the tables where this value can be stored - update it whenever a user uploads or deletes digital objects. This way we don't have to recalculate the value every single time.
  • There is a small risk that, if something is deleted from the filesystem, AtoM wouldn't know. We could potentially make a simple CLI task that would re-run the check and update the value in the DB, to make sure it remains consistent. In most cases this won't be an issue, but it would be a good idea to cover all eventualities.

#6 Updated by Jesús García Crespo over 5 years ago

  • Assignee deleted (Jesús García Crespo)

#7 Updated by Dan Gillean almost 5 years ago

  • Related to Feature #9674: Disable disk usage check when site upload limit is set to unlimited added

#8 Updated by Dan Gillean almost 5 years ago

  • Tracker changed from Bug to Feature
  • Project changed from Access to Memory (AtoM) to AtoM Wishlist
  • Subject changed from Timeout when trying to load Admin > settings page to Improve the upload calculation widget in Admin > Settings so it will not time out when checking large disks
  • Description updated (diff)
  • Category deleted (Performance / scalability)
  • Tested version 2.0.0, 2.0.1, 2.1, 2.1.1, 2.1.2, 2.2, 2.2.1, 2.3 added

Elements of this bug ticket have been addressed by #9275 and #9674, resolving the majority of the immediate issues. Further improvements will require sponsorship for consideration. Correspondingly, I have retooled this ticket as a feature request, updated the details, and am moving it to the AtoM Wishlist project.

#9 Updated by David Juhasz 10 months ago

  • Related to Feature #13411: Add setting to enable or disable repository "upload limit" widget added

Also available in: Atom PDF