[Bug 36277] New: t/db_dependent/api/v1/transfer_limits.t is failing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Bug ID: 36277 Summary: t/db_dependent/api/v1/transfer_limits.t is failing Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Test Suite Assignee: chris@bigballofwax.co.nz Reporter: katrin.fischer@bsz-bw.de QA Contact: testopia@bugs.koha-community.org This failure is from the last run on master: https://jenkins.koha-community.org/job/Koha_Master/ koha_1 | # Inactivity timeout koha_1 | koha_1 | # Failed test 'POST //qBiciZO7stgLDtFSpp1ZivUHaHyn8cMP1d5DXRI:thePassword123@/api/v1/transfer_limits/batch' koha_1 | # at t/db_dependent/api/v1/transfer_limits.t line 210. koha_1 | koha_1 | # Failed test 'SWAGGER3.2.1' koha_1 | # at t/db_dependent/api/v1/transfer_limits.t line 210. koha_1 | # got: undef koha_1 | # expected: '201' koha_1 | # Looks like you failed 2 tests of 26. koha_1 | koha_1 | # Failed test 'batch_add() and batch_delete() tests' koha_1 | # at t/db_dependent/api/v1/transfer_limits.t line 269. koha_1 | # Looks like you failed 1 test of 4. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Passes locally for me. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |RM_priority --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- We have the same in the latest run #2808: koha_1 | # Inactivity timeout koha_1 | koha_1 | # Failed test 'POST //RukUTMflvmiFmWiGbiMJbpcClr5uXXjA4MqgN_LRiINumkvxboZcrYkP:thePassword123@/api/v1/transfer_limits/batch' koha_1 | # at t/db_dependent/api/v1/transfer_limits.t line 210. koha_1 | koha_1 | # Failed test 'SWAGGER3.2.1' koha_1 | # at t/db_dependent/api/v1/transfer_limits.t line 210. koha_1 | # got: undef koha_1 | # expected: '201' koha_1 | # Looks like you failed 2 tests of 26. koha_1 | koha_1 | # Failed test 'batch_add() and batch_delete() tests' -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com --- Comment #3 from Jonathan Druart <jonathan.druart@gmail.com> --- Koha_Master_MDB_Latest/1359 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #4 from Jonathan Druart <jonathan.druart@gmail.com> --- apt install stress stress --cpu 16 (or the number of CPUs you have) Run the following script to inject a lot of libraries use t::lib::TestBuilder; my $builder = t::lib::TestBuilder->new; for my $i ( 1..100000 ) { say $i; $builder->build_object({class => 'Koha::Libraries'}); } % time prove t/db_dependent/api/v1/transfer_limits.t t/db_dependent/api/v1/transfer_limits.t .. 3/4 # Inactivity timeout # Failed test 'POST //hN26548B2h7s3DR9jWZ55OcQnBYvz_olhgm6YAKWU8rRGvCuHOkjM:thePassword123@/api/v1/transfer_limits/batch' # at t/db_dependent/api/v1/transfer_limits.t line 210. # Failed test 'SWAGGER3.2.1' # at t/db_dependent/api/v1/transfer_limits.t line 210. # got: undef # expected: '201' t/db_dependent/api/v1/transfer_limits.t .. 4/4 # Looks like you failed 2 tests of 26. # Failed test 'batch_add() and batch_delete() tests' # at t/db_dependent/api/v1/transfer_limits.t line 269. # Looks like you failed 1 test of 4. t/db_dependent/api/v1/transfer_limits.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/4 subtests Test Summary Report ------------------- t/db_dependent/api/v1/transfer_limits.t (Wstat: 256 Tests: 4 Failed: 1) Failed test: 4 Non-zero exit status: 1 Files=1, Tests=4, 156 wallclock secs ( 0.03 usr 0.00 sys + 105.04 cusr 6.17 csys = 111.24 CPU) Result: FAIL real 2m36.183s user 1m45.121s sys 0m6.208s No idea what to do then... There might be a perf regression somewhere. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #5 from Jonathan Druart <jonathan.druart@gmail.com> --- IMO those 2 nested loops need to be rewritten. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #6 from Jonathan Druart <jonathan.druart@gmail.com> --- 138 foreach my $from ( @from_branches ) { 139 foreach my $to ( @to_branches ) { -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #7 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 163134 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163134&action=edit Bug 36277: Do not fetch the whole library list -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #8 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 163135 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163135&action=edit Bug 36277: Improve algorithmic complexity of batch_add The 2 nested loops are terrible in term of algorithmic complexity. Especially if we are fetching from there. The goal of this patch is to fetch all the limits outside of the loop. If you have 100 libraries, it will remove 100^2 - 1 fetches! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #9 from Jonathan Druart <jonathan.druart@gmail.com> --- This certainly arises because one of the tests does not clean correctly behind it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|chris@bigballofwax.co.nz |jonathan.druart@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #10 from Tomás Cohen Arazi <tomascohen@gmail.com> --- We overlapped here. Nice to see patches, though. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Taking the liberty to speed this one up a bit. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #12 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Katrin Fischer from comment #11)
Taking the liberty to speed this one up a bit.
The patches require proper SO+QA. It's not a trivial change and could introduce regressions. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #13 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (08:41:37) Joubu: cait: I would not push 36277 without a proper review I did some tests, but I'd like to have a proper QA review on these please. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #163134|0 |1 is obsolete| | --- Comment #14 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 163189 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163189&action=edit Bug 36277: Do not fetch the whole library list Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #163135|0 |1 is obsolete| | --- Comment #15 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 163190 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163190&action=edit Bug 36277: Improve algorithmic complexity of batch_add The 2 nested loops are terrible in term of algorithmic complexity. Especially if we are fetching from there. The goal of this patch is to fetch all the limits outside of the loop. If you have 100 libraries, it will remove 100^2 - 1 fetches! Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com QA Contact|testopia@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #16 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Wow, this is really quite a considerable improvement in my opinion. Code is clear to me and tests continue to pass before and after.. and pass consistently after too when I put the system under load now. Thanks Jonathan. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #163189|0 |1 is obsolete| | Attachment #163190|0 |1 is obsolete| | --- Comment #17 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 163199 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163199&action=edit Bug 36277: Do not fetch the whole library list Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #18 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 163200 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163200&action=edit Bug 36277: Improve algorithmic complexity of batch_add The 2 nested loops are terrible in term of algorithmic complexity. Especially if we are fetching from there. The goal of this patch is to fetch all the limits outside of the loop. If you have 100 libraries, it will remove 100^2 - 1 fetches! Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #19 from Tomás Cohen Arazi <tomascohen@gmail.com> --- I tested the UI too, and nothing seems to be broken. I noticed the endpoint allows setting things the UI doesn't, though. But I'll file a separate bug for that. Nice job as usual, Jonathan. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |36329 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36329 [Bug 36329] Transfer limits should respect `BranchTransferLimitsType` -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Failed QA --- Comment #20 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Wait a minute :-D -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Passed QA Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #21 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 163225 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163225&action=edit Bug 36277: Avoid useless warnings As mentioned on bug 36329, this endpoint is only used with one of this parameters: * item_type * collection_code The other will be NULL both on the DB and the (deserialized) request body. For the data from the DB, the author added `|| q{}` but missed to do so on the incoming parameters when generates the hash key. This generates the following warnings when using from the UI: [2024/03/15 11:42:51] [WARN] Use of uninitialized value in sprintf at /kohadevbox/koha/Koha/REST/V1/TransferLimits.pm line 146. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |24.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #22 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to master |Pushed to stable Version(s)|24.05.00 |24.05.00,23.11.04 released in| | --- Comment #23 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 23.11.x for 23.11.04 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #24 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- We still have this one failing seemingly randomly (first it was D11, now on U20) https://jenkins.koha-community.org/job/Koha_Master_U20/lastCompletedBuild/te... :( -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #25 from Jonathan Druart <jonathan.druart@gmail.com> --- The error now starts with "Premature connection close" instead of "Inactivity timeout" koha_1 | # Premature connection close koha_1 | koha_1 | # Failed test 'POST //jJk8CE_fqJgQlLwZpiokcDRYn3YvprTEr5hQLvRFIQjcQDdyCMiW__Dx5yc:thePassword123@/api/v1/transfer_limits/batch' koha_1 | # at t/db_dependent/api/v1/transfer_limits.t line 210. koha_1 | koha_1 | # Failed test 'SWAGGER3.2.1' koha_1 | # at t/db_dependent/api/v1/transfer_limits.t line 210. koha_1 | # got: undef koha_1 | # expected: '201' koha_1 | # Looks like you failed 2 tests of 26. koha_1 | koha_1 | # Failed test 'batch_add() and batch_delete() tests' koha_1 | # at t/db_dependent/api/v1/transfer_limits.t line 269. koha_1 | # Looks like you failed 1 test of 4. koha_1 | [11:07:21] t/db_dependent/api/v1/transfer_limits.t . A successful run on Jenkins (Koha_Master#2813 Docker_15) is slow koha_1 | [10:56:23] t/db_dependent/api/v1/transfer_limits.t ..................................... ok 41681 ms ( 0.01 usr 0.00 sys + 30.81 cusr 1.84 csys = 32.66 CPU) It takes ~10sec on my laptop and I compared with other tests, they are faster on Docker_15. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=36356 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 --- Comment #26 from Jonathan Druart <jonathan.druart@gmail.com> --- Continue on bug 36356 to fix the number of libraries in the DB. But it seems that ~200 libraries is enough to reach the timeout. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable Version(s)|24.05.00,23.11.04 |24.05.00,23.11.04,23.05.10 released in| | CC| |lucas@bywatersolutions.com --- Comment #27 from Lucas Gass <lucas@bywatersolutions.com> --- Backported to 23.05.x for upcoming 23.05.10 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|RM_priority | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|martin.renvoize@ptfs-europe | |.com | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36277 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |wainuiwitikapark@catalyst.n | |et.nz Status|Pushed to oldstable |RESOLVED --- Comment #28 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Not backporting to 22.11 unless requested -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org