Bug #10221

Regenerate derivatives task does not seem to be working

Added by Dan Gillean almost 4 years ago. Updated over 3 years ago.

Status:VerifiedStart date:08/18/2016
Priority:HighDue date:
Assignee:-% Done:

0%

Category:Digital object
Target version:Release 2.4.0
Google Code Legacy ID: Tested version:2.4
Sponsored:No Requires documentation:

Description

I've noticed some issues with this in 2.4, and today was trying to load the demo data into my VM.

After moving all the digital objects into the uploads directory, I ran the digitalobject:regen-derivatives task, cleared my cache, restarted services, and repopulated the search index. When logging into the site however, no thumbnails or reference images show up (showing generic icons instead). Clicking on one reveals the master digital object (whether locally or a linked external master), but I've re-run the task several times and can't get derivs to show up in AtoM. Logged in as an admin so I don't think it's a permissions issue.


Related issues

Related to Access to Memory (AtoM) - Bug #10758: Thumbnails for external DOs do not display in search/brow... Verified 01/11/2017
Related to AtoM Wishlist - Feature #10850: Improve video derivative regeneration process New 02/01/2017

History

#1 Updated by Sara Allain almost 4 years ago

I was also able to reproduce this, with a more conventional workflow:

1. Added a digital object to an item through the UI (tested both a link and an import)
2. Confirmed that the master, reference, and thumbnail were created and visible
3. Ran regen-derivatives task
4. Image replaced by generic placeholder

As with Dan's test, clicking on the placeholder graphic allowed me to view the master image.

#2 Updated by Dan Gillean over 3 years ago

MArking this critical so we fix it before we release 2.4. It's kind of annoying for testing now too.

#3 Updated by David Juhasz over 3 years ago

  • Status changed from New to In progress
  • Assignee set to David Juhasz

#4 Updated by Dan Gillean over 3 years ago

  • Related to Bug #10758: Thumbnails for external DOs do not display in search/browse results added

#5 Updated by David Juhasz over 3 years ago

  • Status changed from In progress to Code Review
  • Assignee changed from David Juhasz to Mike Cantelon

#6 Updated by Mike Cantelon over 3 years ago

  • Status changed from Code Review to Feedback
  • Assignee changed from Mike Cantelon to David Juhasz

The code looks good to me!

#7 Updated by David Juhasz over 3 years ago

  • Status changed from Feedback to QA/Review
  • Assignee changed from David Juhasz to Dan Gillean

Pull request merged to qa/2.4.x

Dan, this is ready for QA/Review.

#8 Updated by Dan Gillean over 3 years ago

  • Status changed from QA/Review to Feedback
  • Assignee changed from Dan Gillean to David Juhasz

I think this is fixed, but I can't totally verify until I know if #10758 is fixed too.

Right now only external thumbnails AND internal ones for video don't seem to be generating. Otherwise, working well.

Working with the demo data locally - none of the video files there got thumbs and they are local uploads. Otherwise, the outstanding issues are covered by #10758.

#9 Updated by Dan Gillean over 3 years ago

Well, just successfully locally uploaded an mp4 and an avi file. Derivatives appeared fine for those during upload, but when i re-ran the task, they were missing. So, something about the task no longer seems to be handling extracting thumbs for videos.

#10 Updated by David Juhasz over 3 years ago

Dan, did you use the type="thumbnail" flag when running the digital object regen task? From the code it looks like the video derivatives will only be generated when the usageId is MASTER_ID or EXTERNAL_URI_ID (ref: https://github.com/artefactual/atom/blob/a6e42164d3b8d4bf3dd0cdb131be4f4277be7478/lib/model/QubitDigitalObject.php#L2043)

#11 Updated by David Juhasz over 3 years ago

  • Assignee changed from David Juhasz to Dan Gillean

#12 Updated by Dan Gillean over 3 years ago

hmmmm i did - but why would it break existing thumbnails then, if they didn't fall under the right type? Will try again with no params.

#13 Updated by David Juhasz over 3 years ago

There are two steps to the script, first it deletes the existing thumbnails, then in generates new thumbnails. The delete works fine, the generate is broken. :( This is true of type="reference" with video files as well.

It's a pretty easy fix to get the reference and thumbnail regeneration fix working for videos, I'll make a new PR.

#14 Updated by David Juhasz over 3 years ago

  • Assignee changed from Dan Gillean to David Juhasz

#15 Updated by David Juhasz over 3 years ago

  • Status changed from Feedback to Code Review
  • Assignee changed from David Juhasz to Mike Cantelon

Mike can you please check out https://github.com/artefactual/atom/pull/497 which should address the video derivative problems Dan describes above.

#16 Updated by Mike Cantelon over 3 years ago

  • Status changed from Code Review to Feedback
  • Assignee changed from Mike Cantelon to David Juhasz

Looks good to me!

#17 Updated by David Juhasz over 3 years ago

  • Status changed from Feedback to QA/Review
  • Assignee changed from David Juhasz to Dan Gillean

I've merged PR497 to qa/2.4.x and it's ready to test again Dan. You can also wait until #10758 is merged to test both together, there's no rush on testing this by itself AFAIK.

A note: when I was testing I wasn't getting any thubmnails generated because I was using videos that were shorter than 30 seconds. It looks like the code currently use a frame from 30s into the video for the thumbnail and just fails if the video is shorter than 30 seconds. :(

#18 Updated by Dan Gillean over 3 years ago

Oh wow! Makes it hard to test, since many longer videos will be more difficult to upload... is this something in AToM's code - e.g. could we potentially lower this value a bit?

#19 Updated by David Juhasz over 3 years ago

Yes, it's hard-coded to a 30 second offset - I was surprised as well. I would be in favour of lowering the value to 5 or 10 seconds. The decision to use 30s was probably based on a client request though, so we should take that into consideration. We could also consider setting a default and having a configuration setting (i.e. in the config files, not UI) to override the default. I would also love to add a check for the total video length to avoid getting a blank thumbnail if the offset is higher than the play time, but adding that functionality would depend on how much time we have available to work on improvements.

#20 Updated by Dan Gillean over 3 years ago

  • Related to Feature #10850: Improve video derivative regeneration process added

#21 Updated by Dan Gillean over 3 years ago

  • Status changed from QA/Review to Verified
  • Assignee deleted (Dan Gillean)

Sounds like the last comments on here are wish list items for the future. Considering this work to be verified for now. I've added a wish-list ticket to capture the ideas mentioned above - see #10850

Also available in: Atom PDF