Feature #11556

Make AtoM's clipboard work with the Varnish Cache web application accelerator

Added by David Hume over 3 years ago. Updated 3 months ago.

Status:QA/ReviewStart date:09/26/2017
Priority:MediumDue date:
Assignee:-% Done:

0%

Category:Clipboard
Target version:Release 2.7.0
Google Code Legacy ID: Tested version:
Sponsored:Yes Requires documentation:

Description

Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy, designed for content-heavy dynamic web sites as well as heavily consumed APIs. Some AtoM sites with much content and frequent visits have incorporated Varnish for noticeable improvements in response.

Around March 2017 a Varnish site was upgraded, and it was noticed clipboard functionality was not working properly.

One of our developers doing a preliminary investigation noted

Looking into it, I think that the clipboard functionality, as-is, is incompatible with Varnish because it relies on sessions to store clipboard items. Sessions rely on cookies and any page requiring a cookie can't be cached by Varnish (there is a way to do per-user caching with Varnish, but it mostly defeats the purpose of caching as you don't get many cache hits). In order to have AtoM have clipboard functionality that'll work with reverse proxies like Varnish (which we'll need for any site with high traffic) we might need to change it so the clipboard stores items in the browser's storage (i.e. LocalStorage).

Other possible solutions suggested:
  • One alternative that I think it's pretty common is to move the clipboard functionality (or any other non cacheable functionality) to XHR calls that are processed asynchronously once the page loads. If you have all the endpoints using a common namespace, e.g. "/clipboard", then you can set up a rule in Varnish to explicitly disable caching for those.
  • Another possibility is to transform the partial components related to the clipboard to ESI components. Symfony 2 brings ESI by default. In Symfony 1 we could use this plugin: http://www.symfony-project.org/plugins/isicsHttpCachePlugin/0_9_1.

Screen Shot 2021-03-23 at 1.51.47 PM.png (178 KB) Steve Breker, 03/23/2021 04:52 PM

History

#1 Updated by David Juhasz about 1 year ago

  • Description updated (diff)

#2 Updated by José Raddaoui Marín 11 months ago

  • Status changed from New to In progress
  • Assignee set to José Raddaoui Marín
  • Target version set to Release 2.7.0
  • Sponsored changed from No to Yes

Initial pull requests to simulate a Varnish setup in the Docker environment for development:

https://github.com/artefactual/atom/pull/1160
https://github.com/artefactual/atom-docs/pull/159

#3 Updated by José Raddaoui Marín 11 months ago

  • Project changed from AtoM Wishlist to Access to Memory (AtoM)
  • Category set to Clipboard

#5 Updated by José Raddaoui Marín 11 months ago

  • Status changed from In progress to Code Review
  • Assignee changed from José Raddaoui Marín to David Juhasz

#6 Updated by José Raddaoui Marín 11 months ago

Some notes for testing/documentation:

  • This has to be tested with and without Varnish and with authenticated and unauthenticated users.
  • My initial research tells me that there is no need to make changes to existing GPDR notices as cookies and local storage are considered the same.
  • At the moment the clipboard does not have TTL. It's stored in the local storage until it's cleared manually. This could be changed to simulate a user session timeout and clear it automatically. It could even be moved to the browser's session storage but then it won't sync between tabs/windows.
  • Login and logout won't clear the clipboard either.
  • Clipboard view page needs to send the slugs from the front-end so it will display "Loading ..." as title until the clipboard is loaded and it will show a red alert with "There was an error loading the clipboard content." as message. I hope you don't see that message so let me know if you want to make changes to it.
  • The clipboard clear by entity button at the bottom of the clipboard page was leading to a confirmation page. This is quite different to the behavior when the clear all link from the header menu is clicked and it complicates the deletion in the new implementation. Since we already have the clear all alert system I decided to do the same for this buttons. If we want to restore the confirmation we should do it for both cases.
  • Existing saved clipboard should still load.
  • The clipboard load form validation has changed and it now shows a "Incorrect clipboard ID and/or action." alert if form validation fails instead of showing "Required" on top of the Clipboard ID field, for example.
  • Export page has been moved from "/object/export" to "/user/clipboardExport".
  • The clipboard content is not checked in the clipboard export page load. Instead of automatically redirecting to the clipboard page an alert will be shown when submitting the form.
  • The clipboard sending message is now shown in an alert instead of redirecting (which is automatically removed on success/error of the sending request).

#7 Updated by David Juhasz 11 months ago

  • Assignee changed from David Juhasz to José Raddaoui Marín

First round of code review done - back to you Radda. :)

#8 Updated by José Raddaoui Marín 11 months ago

Extra notes for testing/documentation:

The clipboard could get out of sync when resources are deleted/edited. This was an issue before but it will be more noticeable now as we don't have a live time for the clipboard at the moment.

After David's feedback, the slugs are validated on clipboard save. That avoids unexpected values in the DB but it also prevents those out of sync slugs to be saved. I've included the count of items saved in the success message to give some sort of indication about what was saved, but the current clipboard is not updated to remove those invalid slugs.

In this case, the validation is needed to secure the save but we could go deeper and use a similar process to keep the clipboard in sync, notifying the user when their clipboard is updated. Alternatively, and probably easier, we could simulate the previous behavior giving a time to live to the clipboard.

#9 Updated by José Raddaoui Marín 11 months ago

  • Assignee changed from José Raddaoui Marín to David Juhasz

#10 Updated by David Juhasz 11 months ago

  • Assignee changed from David Juhasz to José Raddaoui Marín

Thanks for the changes Radda! PR looks great! :D

#11 Updated by José Raddaoui Marín 11 months ago

  • Status changed from Code Review to QA/Review
  • Assignee deleted (José Raddaoui Marín)

Thank you David! Merged in qa/2.x.

To test this changes with Varnish, the POST request to URLs like /clipboard/ must not be cached. An example configuration from the Docker environment can be found in:

https://github.com/artefactual/atom/blob/qa/2.x/docker/etc/varnish/default.vcl

This changes require to upgrade the database. If the work is cherry-picked to an stable branch, download the following script to the AtoM folder:

https://gist.github.com/jraddaoui/d09b397354f14c1a0dfcceb5e9dc04c1

And execute it with:

php symfony tools:run standalone_arMigration0186.php

#12 Updated by Steve Breker 8 months ago

Created fix to update clipboard menu route defaults:

https://github.com/artefactual/atom/pull/1209

This is required for new installs and purges.

Merged to qa/2.x.

#13 Updated by Steve Breker 3 months ago

Adding issue - may only affect devs/testers:

When items have been added to a user's clipboard, and then a tools:purge is completed, the number of items that displays on the clipboard is not updated to zero. The number of items that used to be in the clipboard is still displayed on the icon even though there are zero items in the clipboard. Opening the clipboard will not display anything, but it does not reset the counter either.

See attached screenshot.

Process to replicate:
- enter a test description
- add to clipboard
- run tools:purge
- refresh AtoM - counter will still indicate items in clipboard.

Also available in: Atom PDF