Deadlock issues when running the tests in parallel, and maybe not only in the tests?
Hi :) IIUC this is where the CI calls "prove" to run the tests https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru... And this is the place where KOHA_PROVE_CPUS is set. https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru... It's not set in ".env" file so it should use "nproc" and this calls "prove" with more than one thread. Is that really the case? I tried locally use to "prove -j 4" and I'm getting deadlocks with the database after a few minutes. And do these deadlock in tests mean that Koha can hit them while running in production? I expect the tests to fail due to messing up with each other data but a deadlock is something else. Cheers, -- Victor Grousset/tuxayo
Hey Victor, can you please check whether the $ENV{KOHA_TESTING} variable is set correctly in your CI setup, and if it is not, does it fix the issue? Please see the code in C4::Circulation::SendCirculationAlert(), it is the only place where we do manual db locking as far as I know. It is something we should fix by re-architecturing the message_queue database table for digest messages. However, I'm interested in knowing if the deadlocks now happen because of this and whether it is unit test specific issue because at least based on the comments in SendCirculationAlert() it appears to be so. Regards, Joonas On 25/05/2021 18:45, Victor Grousset/tuxayo wrote:
Hi :)
IIUC this is where the CI calls "prove" to run the tests https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru...
And this is the place where KOHA_PROVE_CPUS is set. https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru...
It's not set in ".env" file so it should use "nproc" and this calls "prove" with more than one thread.
Is that really the case? I tried locally use to "prove -j 4" and I'm getting deadlocks with the database after a few minutes.
And do these deadlock in tests mean that Koha can hit them while running in production?
I expect the tests to fail due to messing up with each other data but a deadlock is something else.
Cheers,
-- Joonas Kylmälä Tietojärjestelmäasiantuntija Kansalliskirjasto Kirjastoverkkopalvelut PL 15 (Unioninkatu 36) 00014 Helsingin yliopisto
Hmm, based on the line https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru... I would think the ENV variable is actually set so maybe it is not this. Any more info on the tables it locks on? Regards, Joonas On 25/05/2021 18:54, Joonas Kylmälä wrote:
Hey Victor,
can you please check whether the $ENV{KOHA_TESTING} variable is set correctly in your CI setup, and if it is not, does it fix the issue? Please see the code in C4::Circulation::SendCirculationAlert(), it is the only place where we do manual db locking as far as I know. It is something we should fix by re-architecturing the message_queue database table for digest messages. However, I'm interested in knowing if the deadlocks now happen because of this and whether it is unit test specific issue because at least based on the comments in SendCirculationAlert() it appears to be so.
Regards, Joonas
On 25/05/2021 18:45, Victor Grousset/tuxayo wrote:
Hi :)
IIUC this is where the CI calls "prove" to run the tests https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru...
And this is the place where KOHA_PROVE_CPUS is set. https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru...
It's not set in ".env" file so it should use "nproc" and this calls "prove" with more than one thread.
Is that really the case? I tried locally use to "prove -j 4" and I'm getting deadlocks with the database after a few minutes.
And do these deadlock in tests mean that Koha can hit them while running in production?
I expect the tests to fail due to messing up with each other data but a deadlock is something else.
Cheers,
-- Joonas Kylmälä Tietojärjestelmäasiantuntija Kansalliskirjasto Kirjastoverkkopalvelut PL 15 (Unioninkatu 36) 00014 Helsingin yliopisto
Hi :) On 21-05-25 18:00, Joonas Kylmälä wrote:
Hmm, based on the line https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru... I would think the ENV variable is actually set so maybe it is not this. Any more info on the tables it locks on?
I did yet processed your previous message but here quickly the error messages. Important info, it has been ran on 19.11.x t/db_dependent/Charset.t ................................................ ok ===( 4190;485 797/? 0/? 17/47 2/15 )============================DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction [for Statement "DELETE FROM issues"] at t/db_dependent/Circulation/Branch.t line 49. ===( 4191;486 797/? 0/? 17/47 3/15 )============================DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction [for Statement "DELETE FROM items"] at t/db_dependent/Circulation.t line 1174. # No tests run! # Failed test 'No tests run for subtest "AllowRenewalIfOtherItemsAvailable tests"' # at t/db_dependent/Circulation.t line 1269. DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction [for Statement "DELETE FROM items"] at t/db_dependent/Circulation.t line 1174. # Looks like your test exited with 255 just after 18. t/db_dependent/Circulation.t ............................................ Dubious, test returned 255 (wstat 65280, 0xff00) -- Victor Grousset/tuxayo
On 26/05/21 3:45 am, Victor Grousset/tuxayo wrote:
Hi :)
IIUC this is where the CI calls "prove" to run the tests https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru...
And this is the place where KOHA_PROVE_CPUS is set. https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru... It's not set in ".env" file so it should use "nproc" and this calls "prove" with more than one thread.
Is that really the case? I tried locally use to "prove -j 4" and I'm getting deadlocks with the database after a few minutes.
hi Victor it is known behaviour with the test suite, i think its caused by the tests mocking database tables in parallel - which is something that would not happen during normal/production usage you can see that all the database tests are forced to run sequentially... prove -j ${KOHA_PROVE_CPUS} --rules='seq=t/db_dependent/**.t' https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/ru...
Thanks folks for the replies. So I was running manually the test suite, and used -j from the CI but didn't used --rules so that's why it was failing. -- Victor Grousset/tuxayo
participants (3)
-
Joonas Kylmälä -
Mason James -
Victor Grousset/tuxayo