Feature #13473

Start integration tests and run them in the CI process

Added by José Raddaoui Marín 10 months ago. Updated 6 months ago.

Status:QA/ReviewStart date:02/09/2021
Priority:MediumDue date:
Assignee:-% Done:

0%

Category:Continuous Integration
Target version:Release 2.7.0
Google Code Legacy ID: Tested version:
Sponsored:No Requires documentation:

Description

The initial idea is to implement this using Cypres and Github Actions:

- https://www.cypress.io/
- https://github.com/features/actions

History

#2 Updated by José Raddaoui Marín 7 months ago

  • Project changed from AtoM Wishlist to Access to Memory (AtoM)
  • Status changed from In progress to Feedback
  • Target version set to Release 2.7.0

The PR was merged a few weeks ago, here some initial feedback about the implementation ...

We are seeing some inconsistencies when these tests are executed in the CI process. In some cases the Chrome browser is not found in the cypress/browsers Docker image (I couldn't find a log) and we've seen the following error in Firefox at least twice already:

  Running:  descriptions_spec.js                                                            (2 of 4)

  Descriptions
    ✓ Creates a basic description (3406ms)
    ✓ Creates a basic description without GUI (1553ms)
    1) Deletes a basic description
    ✓ Deletes a basic description without GUI (1268ms)

  3 passing (9s)
  1 failing

  1) Descriptions
       Deletes a basic description:
     SyntaxError: The following error originated from your application code, not from Cypress.

  > JSON.parse: unexpected character at line 1 column 1 of the JSON data

When Cypress detects uncaught errors originating from your application it will automatically fail the current test.

This behavior is configurable, and you can choose to turn this off by listening to the `uncaught:exception` event.

https://on.cypress.io/uncaught-exception-from-application
  dataFilter@http://localhost/js/fullWidthTreeView.js:110:29
  l/s<@http://localhost/vendor/jquery.js:2:79084
  l@http://localhost/vendor/jquery.js:2:79486
  o/<@http://localhost/vendor/jquery.js:2:82229

In general, using the cypress/browsers Docker image takes quite some time, something that could be improved installing Chrome, Firefox and other deps in the workflow runner directly.

Edit: Chrome error:

Can't run because you've entered an invalid browser name.

Browser: 'chrome' was not found on your system or is not supported by Cypress.

Cypress supports the following browsers:
- chrome
- chromium
- edge
- electron
- firefox

You can also use a custom browser: https://on.cypress.io/customize-browsers

Available browsers found on your system are:
- firefox
- electron

#3 Updated by José Raddaoui Marín 6 months ago

  • Category set to Continuous Integration
  • Status changed from Feedback to In progress

WIP PR trying to improve the consistency of the e2e tests:

https://github.com/artefactual/atom/pull/1319

#4 Updated by José Raddaoui Marín 6 months ago

  • Status changed from In progress to Code Review

It has been a while since I don't see the Firefox error and I can't find a way to reproduce it locally. The PR from the previous update will use the Chrome and Firefox browsers installed in the runner instead of the ones from a Cypress Docker image, this speed ups the process (as it avoids downloading that image and allows to cache the NPM deps) and most likely avoids the "Browser: 'chrome' was not found" issue. Thanks to a fix from Mike C. (related to the new culture cookie), we have been able to remove a hack from the test code. So hopefully that PR improves the situation.

#5 Updated by José Raddaoui Marín 6 months ago

  • Status changed from Code Review to QA/Review
  • Assignee deleted (José Raddaoui Marín)

Just before merging this PR I saw the Firefox error again in another PR (with the previous implementation). Lets see how it goes after merging these changes.

Also available in: Atom PDF