Feature #10309

Generate archival description reports asynchronously

Added by Mike Gale almost 4 years ago. Updated almost 3 years ago.

Status:VerifiedStart date:05/02/2016
Priority:MediumDue date:
Assignee:-% Done:

0%

Category:ReportsEstimated time:70.00 hours
Target version:Release 2.4.0
Google Code Legacy ID: Tested version:
Sponsored:Yes Requires documentation:

Description

AtoM includes several basic report options available from the archival description view page: file report, item report, CSV box labels, and a storage locations report. These reports are currently being generated synchronously on-demand, via the web browser. Because of this, they can often time out during larger requests (e.g. when the descriptive hierarchy is larger).

This feature will move the generation of these reports to the job scheduler, so they can be performed asynchronously in the background, thereby avoiding browser timeouts. We will follow a similar approach to how the Finding aid generation is handled in the user interface, making a link available to users on the current page after the report is generated. This way, public users will still have access to the public reports (file and item reports) without having to gain access to the Jobs page.

For download, users will have the option of generating the report in CSV or HTML. for the HTML version, users will still download the file, and view it locally with a browser.

History

#2 Updated by Mike Gale over 3 years ago

NOTE: This feature requires the user to run php symfony tools:upgrade-sql

#3 Updated by Mike Gale over 3 years ago

NOTE: there's a new setting on the admin settings' global form, whether to generate reports as public user or not. This determines if "Retrieval information" will show up in item/file list reports. Box label and physical storage location reports should be hidden from non-authenticated users in the new 'Existing reports' section of the reports page for archival descriptions regardless of settings (same as old behavior).

#4 Updated by Mike Gale over 3 years ago

  • Status changed from New to QA/Review
  • Assignee changed from Mike Gale to Dan Gillean

#5 Updated by Dan Gillean over 3 years ago

  • Category set to Reports
  • Target version set to Release 2.4.0
  • Sponsored changed from No to Yes

#6 Updated by Dan Gillean about 3 years ago

  • Assignee deleted (Dan Gillean)

#7 Updated by Kelly Stewart about 3 years ago

  • Assignee set to Kelly Stewart

As anonymous, I get a 500 error when trying to generate an item report in any format (CSV or HTML) on atomqa24x

#8 Updated by Kelly Stewart about 3 years ago

  • Assignee changed from Kelly Stewart to Nick Wilkinson

#9 Updated by Kelly Stewart about 3 years ago

  • Assignee changed from Nick Wilkinson to Kelly Stewart

#10 Updated by Kelly Stewart about 3 years ago

  • Assignee changed from Kelly Stewart to Nick Wilkinson

As anonymous user, generated an item-level report of the Ada Lovelace fonds, which has 1 item in quotation marks. Quotation marks in the item list report become “

atomqa24x

#11 Updated by Kelly Stewart about 3 years ago

  • Requires documentation changed from Yes to No

#12 Updated by Dan Gillean about 3 years ago

  • Requires documentation changed from No to Yes

These are the pages that need to be updated for docs on this feature:

There are updates to the settings page as well - see Mike's comment 8 above.

Some of the changes that need to be captured include:
  • Reports can be generated in either CSV or HTML to be downloaded now (previously it was HTML served in the browser only)
  • Once a report is generated, it remains available - public users access pre-generated file/item reports, but only authenticated users can generate them (if I recall correctly)
  • The intro on the page should mention that this is being handled by the job scheduler, meaning users must ensure that the atom-worker is running for report generation to work. See for example: https://github.com/artefactual/atom-docs/blob/2.4/user-manual/import-export/import-xml.rst

etc...

#13 Updated by Nick Wilkinson about 3 years ago

  • Status changed from QA/Review to In progress
  • Assignee changed from Nick Wilkinson to José Raddaoui Marín

#14 Updated by José Raddaoui Marín about 3 years ago

  • Status changed from In progress to Code Review
  • Assignee changed from José Raddaoui Marín to Nick Wilkinson

Kelly Stewart wrote:

As anonymous user, generated an item-level report of the Ada Lovelace fonds, which has 1 item in quotation marks. Quotation marks in the item list report become “

atomqa24x

It should be fixed in https://github.com/artefactual/atom/pull/582

#15 Updated by Nick Wilkinson about 3 years ago

  • Assignee changed from Nick Wilkinson to Steve Breker

Hi Steve, assigning this to you for CR.

#16 Updated by José Raddaoui Marín about 3 years ago

  • Status changed from Code Review to QA/Review
  • Assignee changed from Steve Breker to Nick Wilkinson

Already reviewed, merged in qa/2.4.x. Not sure if this needs to be deployed somewhere or tested directly.

#17 Updated by Nick Wilkinson about 3 years ago

  • Assignee deleted (Nick Wilkinson)

To be tested for the 2.4 release.

#18 Updated by Dan Gillean almost 3 years ago

  • Status changed from QA/Review to Feedback
  • Assignee set to José Raddaoui Marín

Issue 1: if dates of creation are removed or changed to dates of accumulation, they are not removed from the resulting report, even if you regenerate the reports

I'm working in the most recent qa/2.4.x branch in a local Vagrant box, with the AtoM demo data set loaded - using the "Irving Steinberg Sudbury Slide Collection" for testing. To reproduce:

  • Navigate to an item record and enter edit mode
  • Change the date type to accumulation and save
  • In a different item-level record, delete the current date of creation and add a date of accumulation instead, and save
  • Generate the reports in both HTML and CSV formats
Resulting error
  • Both report formats still display the outdated dates of creation information
  • I tried clearing the cache, restarting memcached and php-fpm, and repeating the steps above, but got the same report
  • Either the original reports are not being replaced by updated ones, or the report generation is somehow using out-of-date information.
Expected result
As far as I know, the item and file reports are only configured to display dates of creation - the display date and the startDate are included. Accumulation dates normally don't show up. I would therefore expect that:
  • Where there is only a date of accumulation, then no date is included in the updated report generated
  • Where there is both a date of accumulation and a date of creation on the item, then only the date of creation will be displayed

Ideal result
The file and item list reports should use the same event columns used by the CSV import/export - so that multiple dates, and different event types, can be included in the output. Columns should include: eventTypes, eventDates, eventStartDates, and eventEndDates.


Issue 2: Digital object thumbnails are broken in the HTML reports generated. Testing was done on Chrome 59.0.3071.115.

#19 Updated by José Raddaoui Marín almost 3 years ago

  • Assignee changed from José Raddaoui Marín to Nick Wilkinson

#20 Updated by Nick Wilkinson almost 3 years ago

  • Assignee changed from Nick Wilkinson to Mike Gale

Hi Mike, reassigning this to you -- Radda thinks you'll have more insight into this than him.

#21 Updated by Mike Gale almost 3 years ago

Issue 1 in Dan's comment #18 was actually due to something unexpected. It seems the code, if it can't find a creation date for an item level description, will go up the hierarchy until it finds a parent with a creation date, and fill that in as the item's creation date as well. In Dan's case the parents' creation date was the same as the items' previous creation dates, hence why he thought there was outdated info in there.

I discussed with Dan and we're going to remove this functionality and have the items display blank for creation dates if they don't have one.

#22 Updated by Mike Gale almost 3 years ago

  • Status changed from Feedback to Code Review
  • Assignee changed from Mike Gale to José Raddaoui Marín

#23 Updated by José Raddaoui Marín almost 3 years ago

  • Status changed from Code Review to Feedback
  • Assignee changed from José Raddaoui Marín to Mike Gale

#24 Updated by Mike Gale almost 3 years ago

  • Status changed from Feedback to QA/Review
  • Assignee changed from Mike Gale to Dan Gillean

Merged qa/2.4.x 0e9282f9662e9bff29410307393bdb5b77d83ca2 & 17d901b36bf57a460fe333caf6b84487cb23da3a

This fix requires a tools:sql-upgrade.

#25 Updated by Dan Gillean almost 3 years ago

  • Status changed from QA/Review to Feedback
  • Assignee changed from Dan Gillean to Mike Gale

Looks good, but I still can't see the thumbnails (issue 2 in note 18). Any thoughts?

#26 Updated by Dan Gillean almost 3 years ago

  • Status changed from Feedback to Verified
  • Assignee deleted (Mike Gale)

Ok, that was my bad.

For the thumbs, the code for the reports builds the URLs to the thumbs by using the Base URL available in Admin > Settings > Site information.

In my case, I'm using a copy of the demo data as my test corpus, so it was set to demo.accesstomemory.org - though my local vagrant box is actually at 10.10.10.10. When I updated the base URL setting, the problem was resolved - thumbnails now appear in the HTML as expected.

Thanks Mike, and sorry for the red herring!

#27 Updated by Dan Gillean almost 3 years ago

  • Requires documentation deleted (Yes)

Also available in: Atom PDF