Make AtoM's clipboard work with the Varnish Cache web application accelerator
|Target version:||Release 2.7.0|
|Google Code Legacy ID:||Tested version:|
Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy, designed for content-heavy dynamic web sites as well as heavily consumed APIs. Some AtoM sites with much content and frequent visits have incorporated Varnish for noticeable improvements in response.
Around March 2017 a Varnish site was upgraded, and it was noticed clipboard functionality was not working properly.
One of our developers doing a preliminary investigation noted
Other possible solutions suggested:
Looking into it, I think that the clipboard functionality, as-is, is incompatible with Varnish because it relies on sessions to store clipboard items. Sessions rely on cookies and any page requiring a cookie can't be cached by Varnish (there is a way to do per-user caching with Varnish, but it mostly defeats the purpose of caching as you don't get many cache hits). In order to have AtoM have clipboard functionality that'll work with reverse proxies like Varnish (which we'll need for any site with high traffic) we might need to change it so the clipboard stores items in the browser's storage (i.e. LocalStorage).
- One alternative that I think it's pretty common is to move the clipboard functionality (or any other non cacheable functionality) to XHR calls that are processed asynchronously once the page loads. If you have all the endpoints using a common namespace, e.g. "/clipboard", then you can set up a rule in Varnish to explicitly disable caching for those.
- Another possibility is to transform the partial components related to the clipboard to ESI components. Symfony 2 brings ESI by default. In Symfony 1 we could use this plugin: http://www.symfony-project.org/plugins/isicsHttpCachePlugin/0_9_1.
#2 Updated by José Raddaoui Marín almost 2 years ago
- Status changed from New to In progress
- Assignee set to José Raddaoui Marín
- Target version set to Release 2.7.0
- Sponsored changed from No to Yes
#6 Updated by José Raddaoui Marín almost 2 years ago
Some notes for testing/documentation:
- This has to be tested with and without Varnish and with authenticated and unauthenticated users.
- My initial research tells me that there is no need to make changes to existing GPDR notices as cookies and local storage are considered the same.
- At the moment the clipboard does not have TTL. It's stored in the local storage until it's cleared manually. This could be changed to simulate a user session timeout and clear it automatically. It could even be moved to the browser's session storage but then it won't sync between tabs/windows.
- Login and logout won't clear the clipboard either.
- Clipboard view page needs to send the slugs from the front-end so it will display "Loading ..." as title until the clipboard is loaded and it will show a red alert with "There was an error loading the clipboard content." as message. I hope you don't see that message so let me know if you want to make changes to it.
- The clipboard clear by entity button at the bottom of the clipboard page was leading to a confirmation page. This is quite different to the behavior when the clear all link from the header menu is clicked and it complicates the deletion in the new implementation. Since we already have the clear all alert system I decided to do the same for this buttons. If we want to restore the confirmation we should do it for both cases.
- Existing saved clipboard should still load.
- The clipboard load form validation has changed and it now shows a "Incorrect clipboard ID and/or action." alert if form validation fails instead of showing "Required" on top of the Clipboard ID field, for example.
- Export page has been moved from "/object/export" to "/user/clipboardExport".
- The clipboard content is not checked in the clipboard export page load. Instead of automatically redirecting to the clipboard page an alert will be shown when submitting the form.
- The clipboard sending message is now shown in an alert instead of redirecting (which is automatically removed on success/error of the sending request).
#8 Updated by José Raddaoui Marín almost 2 years ago
Extra notes for testing/documentation:
The clipboard could get out of sync when resources are deleted/edited. This was an issue before but it will be more noticeable now as we don't have a live time for the clipboard at the moment.
After David's feedback, the slugs are validated on clipboard save. That avoids unexpected values in the DB but it also prevents those out of sync slugs to be saved. I've included the count of items saved in the success message to give some sort of indication about what was saved, but the current clipboard is not updated to remove those invalid slugs.
In this case, the validation is needed to secure the save but we could go deeper and use a similar process to keep the clipboard in sync, notifying the user when their clipboard is updated. Alternatively, and probably easier, we could simulate the previous behavior giving a time to live to the clipboard.
#11 Updated by José Raddaoui Marín almost 2 years ago
- Status changed from Code Review to QA/Review
- Assignee deleted (
José Raddaoui Marín)
Thank you David! Merged in qa/2.x.
To test this changes with Varnish, the POST request to URLs like
/clipboard/ must not be cached. An example configuration from the Docker environment can be found in:
This changes require to upgrade the database. If the work is cherry-picked to an stable branch, download the following script to the AtoM folder:
And execute it with:
php symfony tools:run standalone_arMigration0186.php
#13 Updated by Steve Breker about 1 year ago
- File Screen Shot 2021-03-23 at 1.51.47 PM.png added
Adding issue - may only affect devs/testers:
When items have been added to a user's clipboard, and then a tools:purge is completed, the number of items that displays on the clipboard is not updated to zero. The number of items that used to be in the clipboard is still displayed on the icon even though there are zero items in the clipboard. Opening the clipboard will not display anything, but it does not reset the counter either.
See attached screenshot.
Process to replicate:
- enter a test description
- add to clipboard
- run tools:purge
- refresh AtoM - counter will still indicate items in clipboard.
#15 Updated by Dan Gillean 7 months ago
This moves the clipboard items to browser storage - meaning a user can log out/in and still keep clipboard results. The existing documentation should be reviewed and updated as needed with this in mind.
Additionally, not sure if Varnish support is documented anywhere at present.
should check what happens if you delete a description that is added to the clipboard - it should trigger an update to the browser storage, but may not currently.