[Bug 26641] New: link_bibs_to_authorities.pl: Add the ability to specify the MARC field to operate on
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Bug ID: 26641 Summary: link_bibs_to_authorities.pl: Add the ability to specify the MARC field to operate on Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: a.roussos@dataly.gr QA Contact: testopia@bugs.koha-community.org CC: robin@catalyst.net.nz When running link_bibs_to_authorities.pl, it could be useful to have the ability to specify which MARC field the script should operate on. For example, when you have a catalog where most links are in place, and you don't want the script to process every MARC field but rather limit it to a specific MARC field (e.g. 700) because you've observed that there are many missing links for that particular MARC field. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |a.roussos@dataly.gr |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 --- Comment #1 from Andreas Roussos <a.roussos@dataly.gr> --- I've conducted the following tests with a modified version of the script that accepts the MARC field to operate on as an argument (my Koha DB has approximately 22k biblio records and 7k authority records): Command line used Time (MM:SS) Comments ----------------------------------------------- ------------ -------- link_bibs_to_authorities.pl -v -l --test 05:49 1st run link_bibs_to_authorities.pl -v -l --test 06:07 2nd run link_bibs_to_authorities.pl -v -l --test 06:40 3rd run Command line used Time (MM:SS) Comments ----------------------------------------------- ------------ -------- link_bibs_to_authorities.pl -v -l --test -g=700 03:02 1st run link_bibs_to_authorities.pl -v -l --test -g=710 00:58 1st run link_bibs_to_authorities.pl -v -l --test -g=710 00:58 2nd run -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Medium patch Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 --- Comment #2 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 111395 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111395&action=edit Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl When running link_bibs_to_authorities.pl, it could be useful to have the ability to specify which MARC field the script should operate on. For example, when you have a catalog where most links are in place, and you don't want the script to process every MARC field but rather limit it to a specific MARC field (e.g. 700) because you've observed that there are many missing links for that particular MARC field. This patch provides that enhancement. Test plan: 1) Run the script as follows (preferably in a test DB): time link_bibs_to_authorities.pl -v -l --test ... and notice how long it takes to complete. 2) Apply the patch. 3) Run the script again, this time providing the MARC field to work on: time link_bibs_to_authorities.pl -v -l --test -g=700 ... and, again, notice how long it takes to complete. The 2nd run should run faster than the 1st one. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #3 from David Cook <dcook@prosentient.com.au> --- You must have been reading my mind last week, because I was just about to start work on something similar. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- Comment on attachment 111395 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111395 Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl Review of attachment 111395: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=26641&attachment=111395) ----------------------------------------------------------------- ::: misc/link_bibs_to_authorities.pl @@ +200,4 @@
}
my $frameworkcode = GetFrameworkCode($biblionumber); + my $allowrelink = C4::Context->preference("CatalogModuleRelink") || '';
From a performance (and reliability) perspective, this lookup should be done outside the function and before the while loop.
I can submit a follow-up patch for that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- Actually, looks like we had something different in mind, but that's OK. I'll still work on this anyway. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #111395|0 |1 is obsolete| | --- Comment #6 from David Cook <dcook@prosentient.com.au> --- Created attachment 111461 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111461&action=edit Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl When running link_bibs_to_authorities.pl, it could be useful to have the ability to specify which MARC field the script should operate on. For example, when you have a catalog where most links are in place, and you don't want the script to process every MARC field but rather limit it to a specific MARC field (e.g. 700) because you've observed that there are many missing links for that particular MARC field. This patch provides that enhancement. Test plan: 1) Run the script as follows (preferably in a test DB): time link_bibs_to_authorities.pl -v -l --test ... and notice how long it takes to complete. 2) Apply the patch. 3) Run the script again, this time providing the MARC field to work on: time link_bibs_to_authorities.pl -v -l --test -g=700 ... and, again, notice how long it takes to complete. The 2nd run should run faster than the 1st one. Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 --- Comment #7 from David Cook <dcook@prosentient.com.au> --- Created attachment 111462 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111462&action=edit Bug 26641: Optimize CatalogModuleRelink lookup This patch moves the CatalogModuleRelink lookup outside the loop to improve performance (especially for large databases), and includes a tester friendly test plan. Test plan: 0. Using koha-testing-docker 1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev 2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked 3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev 4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked 5. Note output appear to all be names (rather than subjects or other authorities) 6. Note also that the 2nd run with -g 700 is almost 2x faster than without -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Andreas, if you test and sign off my patch, you can move this to "Signed Off". -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #111462|0 |1 is obsolete| | --- Comment #9 from David Cook <dcook@prosentient.com.au> --- Created attachment 111463 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111463&action=edit Bug 26641: Optimize CatalogModuleRelink lookup This patch moves the CatalogModuleRelink lookup outside the loop to improve performance (especially for large databases), and includes a tester friendly test plan. Test plan: 0. Using koha-testing-docker 1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev 2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked 3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev 4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked 5. Note output appear to all be names (rather than subjects or other authorities) 6. Note also that the 2nd run with -g 700 is almost 2x faster than without -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=21828 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 George Veranis <gveranis@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gveranis@dataly.gr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 --- Comment #10 from Andreas Roussos <a.roussos@dataly.gr> --- (In reply to David Cook from comment #8)
Andreas, if you test and sign off my patch, you can move this to "Signed Off". Thank you for your interest in this bug report, David. I've set up a koha-testing-docker development environment and will now go through your test plan.
-- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #111463|0 |1 is obsolete| | --- Comment #11 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 111829 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111829&action=edit Bug 26641: Optimize CatalogModuleRelink lookup This patch moves the CatalogModuleRelink lookup outside the loop to improve performance (especially for large databases), and includes a tester friendly test plan. Test plan: 0. Using koha-testing-docker 1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev 2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked 3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev 4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked 5. Note output appear to all be names (rather than subjects or other authorities) 6. Note also that the 2nd run with -g 700 is almost 2x faster than without Signed-off-by: Andreas Roussos <a.roussos@dataly.gr> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #111461|0 |1 is obsolete| | --- Comment #12 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 111868 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111868&action=edit Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl When running link_bibs_to_authorities.pl, it could be useful to have the ability to specify which MARC field the script should operate on. For example, when you have a catalog where most links are in place, and you don't want the script to process every MARC field but rather limit it to a specific MARC field (e.g. 700) because you've observed that there are many missing links for that particular MARC field. This patch provides that enhancement. Test plan: 1) Run the script as follows (preferably in a test DB): time link_bibs_to_authorities.pl -v -l --test ... and notice how long it takes to complete. 2) Apply the patch. 3) Run the script again, this time providing the MARC field to work on: time link_bibs_to_authorities.pl -v -l --test -g=700 ... and, again, notice how long it takes to complete. The 2nd run should run faster than the 1st one. Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #111829|0 |1 is obsolete| | --- Comment #13 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 111869 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=111869&action=edit Bug 26641: Optimize CatalogModuleRelink lookup This patch moves the CatalogModuleRelink lookup outside the loop to improve performance (especially for large databases), and includes a tester friendly test plan. Test plan: 0. Using koha-testing-docker 1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev 2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked 3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev 4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked 5. Note output appear to all be names (rather than subjects or other authorities) 6. Note also that the 2nd run with -g 700 is almost 2x faster than without Signed-off-by: Andreas Roussos <a.roussos@dataly.gr> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |martin.renvoize@ptfs-europe | |.com --- Comment #14 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Love the teamwork here! All seems to work as expected and I'm not seeing any regressions. QA Scripts happy.. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #15 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- why "g"? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |20.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 --- Comment #16 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 20.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com --- Comment #17 from Lucas Gass <lucas@bywatersolutions.com> --- enhancement will not be backported to 20.05.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- It's a shame that we didn't make this a repeatable option... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 --- Comment #19 from Andreas Roussos <a.roussos@dataly.gr> --- (In reply to Jonathan Druart from comment #15)
why "g"? Because "t" was already taken (for the -t/--test option).
Apologies for necro posting ;-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26641 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=34815 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org