[Bug 23377] New: bulkmarcimport.pl disables syspref caching
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Bug ID: 23377 Summary: bulkmarcimport.pl disables syspref caching Change sponsored?: --- Product: Koha Version: 18.11 Hardware: All OS: All Status: NEW Severity: minor Priority: P5 - low Component: Tools Assignee: koha-bugs@lists.koha-community.org Reporter: ian@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org bulkmarcimport.pl disables syspref caching in order to explicitly set CataloguingLog and AuthoritiesLog to false (in order to boost performance, according to the notes). However, doing so results in numerous other syspref lookups over the course of the script, increasing runtime by about 33%. Sysprefs looked up during a plain, no-options import include: 1. DefaultClassificationSource 2. TimeFormat 3. dateformat 4. CataloguingLog 5. BiblioAddsAuthorities 6. marcflavour 7. SearchEngine 8. OAI-PMH:AutoUpdateSets While disabling additional logging on import is a good thing for performance, these additional DB queries offset that gain. Using the ENV variable override option may be a better way to temporarily disable these logs for the course of the script, whilst still keeping all the other sysprefs cached. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 --- Comment #1 from Ian Walls <ian@bywatersolutions.com> --- Created attachment 91898 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=91898&action=edit Bug 23377: use ENV syspref override instead of complete disable In an attempt to save time, bulkmarcimport temporarily sets CataloguingLog and AuthoritiesLog to 0. It does this by disabling syspref caching and saving the changes to the database (then replacing the original values at completion). Unfortunately, this disables other key sysprefs from being cached, and results in a 50% increase in processing time for the script. This patch instead utilizes the ENV variable override feature of sysprefs, which preempts the cache in C4::Context->preference(). To test: 1. Perform a bulkmarcimport with a reasonable number of biblios (~1000 will do) 2. Note the time taken to complete 3. Apply patch 4. Revert the biblio load performed 5. Perform another bulkmarcimport with the same biblios and commandline options 6. Note the time taken to complete 7. Compare times. The time from step 6 should be about 33% less than the time from step 2 8. Check Cataloguing and Authorities Logs to verify imported records were not logged 9. Profit! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Ian Walls <ian@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de Assignee|koha-bugs@lists.koha-commun |ian@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |black23@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #91898|0 |1 is obsolete| | --- Comment #2 from Michal Denar <black23@gmail.com> --- Created attachment 97143 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=97143&action=edit Bug 23377: use ENV syspref override instead of complete disable In an attempt to save time, bulkmarcimport temporarily sets CataloguingLog and AuthoritiesLog to 0. It does this by disabling syspref caching and saving the changes to the database (then replacing the original values at completion). Unfortunately, this disables other key sysprefs from being cached, and results in a 50% increase in processing time for the script. This patch instead utilizes the ENV variable override feature of sysprefs, which preempts the cache in C4::Context->preference(). To test: 1. Perform a bulkmarcimport with a reasonable number of biblios (~1000 will do) 2. Note the time taken to complete 3. Apply patch 4. Revert the biblio load performed 5. Perform another bulkmarcimport with the same biblios and commandline options 6. Note the time taken to complete 7. Compare times. The time from step 6 should be about 33% less than the time from step 2 8. Check Cataloguing and Authorities Logs to verify imported records were not logged 9. Profit! Signed-off-by: Michal Denar <black23@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl Status|Signed Off |BLOCKED --- Comment #3 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Looking here -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|BLOCKED |Passed QA Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #97143|0 |1 is obsolete| | --- Comment #4 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 97183 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=97183&action=edit Bug 23377: use ENV syspref override instead of complete disable In an attempt to save time, bulkmarcimport temporarily sets CataloguingLog and AuthoritiesLog to 0. It does this by disabling syspref caching and saving the changes to the database (then replacing the original values at completion). Unfortunately, this disables other key sysprefs from being cached, and results in a 50% increase in processing time for the script. This patch instead utilizes the ENV variable override feature of sysprefs, which preempts the cache in C4::Context->preference(). To test: 1. Perform a bulkmarcimport with a reasonable number of biblios (~1000 will do) 2. Note the time taken to complete 3. Apply patch 4. Revert the biblio load performed 5. Perform another bulkmarcimport with the same biblios and commandline options 6. Note the time taken to complete 7. Compare times. The time from step 6 should be about 33% less than the time from step 2 8. Check Cataloguing and Authorities Logs to verify imported records were not logged 9. Profit! Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 --- Comment #5 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 97184 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=97184&action=edit Bug 23377: (QA follow-up) Use OVERRIDE_SYSPREF SYSPREF_OVERRIDE unfortunately is not supported ;) Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 --- Comment #6 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Good to see you back, Ian. Ready to QA again :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | Version|18.11 |master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |20.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 --- Comment #7 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work everyone! Pushed to master for 20.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Joy Nelson <joy@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joy@bywatersolutions.com Version(s)|20.05.00 |20.05.00, 19.11.03 released in| | Status|Pushed to master |Pushed to stable --- Comment #8 from Joy Nelson <joy@bywatersolutions.com> --- Pushed to 19.11.x branch for 19.11.03 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|20.05.00, 19.11.03 |20.05.00, 19.11.03, released in| |19.05.08 CC| |lucas@bywatersolutions.com Status|Pushed to stable |Pushed to oldstable --- Comment #9 from Lucas Gass <lucas@bywatersolutions.com> --- backported to 19.05.x for 19.05.08 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23377 Hayley Mapley <hayleymapley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hayleymapley@catalyst.net.n | |z --- Comment #10 from Hayley Mapley <hayleymapley@catalyst.net.nz> --- Enhancement will not be backported to 18.11.x series. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org