[Bug 42849] New: Retry failed Cypress specs in CI to reduce failures from flaky tests
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 Bug ID: 42849 Summary: Retry failed Cypress specs in CI to reduce failures from flaky tests Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: kyle@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org Target Milestone: --- When we run cypress tests, they sometimes fail but pass when re-run. This appears to simply be a timing issue where the reponse is too slow. Allowing for a couple retries should previous spurious failures! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |minor Assignee|koha-bugs@lists.koha-commun |kyle@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 --- Comment #1 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200481 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200481&action=edit Bug 42849: Retry failed Cypress specs in CI to reduce failures from flaky tests When we run cypress tests, they sometimes fail but pass when re-run. This appears to simply be a timing issue where the response is too slow. Allowing for a couple retries should previous spurious failures! Test plan: 1) Run cypress tests 2) Tests (should) pass 3) Apply this patch 4) Run tests again 5) Note they (still) pass! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200481|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 --- Comment #2 from David Nind <david@davidnind.com> --- Created attachment 200713 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200713&action=edit Bug 42849: Retry failed Cypress specs in CI to reduce failures from flaky tests When we run cypress tests, they sometimes fail but pass when re-run. This appears to simply be a timing issue where the response is too slow. Allowing for a couple retries should previous spurious failures! Test plan: 1) Run cypress tests 2) Tests (should) pass 3) Apply this patch 4) Run tests again 5) Note they (still) pass! Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #3 from David Nind <david@davidnind.com> --- bookingsModalDatePicker_spec.ts fails before and after the patch. With the patch, it attempts the test 3 times before moving onto the next test. The failing test is fixed by Bug 42862. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com --- Comment #4 from Jonathan Druart <jonathan.druart@gmail.com> --- Which flaky tests are we trying to fix? Usually the tests that fail are not correctly written. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 --- Comment #5 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- (In reply to Jonathan Druart from comment #4)
Which flaky tests are we trying to fix? Usually the tests that fail are not correctly written.
Good question. This was my first fix attempt for bookingsModalDatePicker_spec.ts, with the solution being bug 42862. I think this is still a good patch which addresses issues related to the hardware the tests are being run on rather than our code, but it doesn't address a specific failure at this time. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lisette@bywatersolutions.co | |m QA Contact|testopia@bugs.koha-communit |victor@tuxayo.net |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |arthur.suzuki@biblibre.com QA Contact|victor@tuxayo.net |arthur.suzuki@biblibre.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200713|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 --- Comment #6 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- Created attachment 209653 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209653&action=edit Bug 42849: Retry failed Cypress specs in CI to reduce failures from flaky tests When we run cypress tests, they sometimes fail but pass when re-run. This appears to simply be a timing issue where the response is too slow. Allowing for a couple retries should previous spurious failures! Test plan: 1) Run cypress tests 2) Tests (should) pass 3) Apply this patch 4) Run tests again 5) Note they (still) pass! Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 --- Comment #7 from Jonathan Druart <jonathan.druart@gmail.com> --- Without clear examples and reasons I don't think we should push this. We really don't have much failures that would be fixed by this change. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42849 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Failed QA CC| |pedro.amorim@openfifth.co.u | |k --- Comment #8 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Hi all, deciding not to push this. Reasons: - Kyle's comment points to bug 42862 as being the solution for what originally motivated this. - David's comment stating that bookingsModalDatePicker_spec.ts fails before and after the patch (retries don't help). - Joubu's reasoning on the lack of a clear, reproducible issue. If this does help with failures on slow hardware, we need to be able to show it, e.g. CI runs where a test failed and then passed on retry, or a test that fails reliably on a slowed down machine. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org