[Bug 28011] New: Add CLI script to update bibs from linked authority records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 Bug ID: 28011 Summary: Add CLI script to update bibs from linked authority records Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org CC: robin@catalyst.net.nz The script link_bibs_to_authorities.pl will link bibs to authorities, but it won't update the bibs with the data from the authority. I'm proposing adding a new script which will update the bib records from its linked authorities. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=24501 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 Henry Bolshaw <bolshawh@parliament.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bolshawh@parliament.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- This is still on my TODO short list but still probably a little ways off as I have higher priorities at the moment. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- It seems like there's a few ways this could be done. One idea is to fetch every bib record that contains at least one $9 linkage and then compiling a list of unique IDs for authority records. Then I'll run the following: my $record = GetAuthority($authid); merge({ mergefrom => $authid, MARCfrom => $record, mergeto => $authid, MARCto => $record }); But that will involve double-handling bib records. -- Alternatively, I suppose I could fetch every bib record that contains at least one $9 linkage, and then run the following for each unique linkage: my $record = GetAuthority($authid); merge({ mergefrom => $authid, MARCfrom => $record, mergeto => $authid, MARCto => $record, biblionumbers => [$biblionumber] }); I could cache the $record object to reduce some database fetches, except that merge() is super inefficient. -- I suppose I could write a custom function... It looks like the heading data is obtained using the following: "@record_to = $MARCto->field($auth_tag_to_report_to)->subfields()" Then create $field_to from @record_to, and then $field->replace_with($field_to). Of course, writing a new function adds complications, especially if I don't want to refactor merge(), which is an intimidating task. Maybe I will just maintain this locally as that is less risky. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Actually, I'd say it's probably better to just be inefficient than have a custom implementation. The simplest version is just to compile a list of unique authority IDs and then run the following: my $record = GetAuthority($authid); merge({ mergefrom => $authid, MARCfrom => $record, mergeto => $authid, MARCto => $record }); It will be inefficient but it will be a CLI script which can just run as long as it needs to. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- Created attachment 121405 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=121405&action=edit Bug 28011: Add CLI script to update bib headings from linked auth records This patch adds a script which updates bib headings from authority records. This helps normalize the headings to remove differences in punctuation, capitalization, etc. This can be useful to run after link_bibs_to_authorities.pl. Test plan: 1. Apply the patch 2. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=192 3. Mangle the HTML to remove the accents from 100$a and save the record 4. Note the bib record now says "By: O Cadhain, Mairtin" without the accents 5. Run the script 6. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=192 7. Note that the accents have been restored and it now says "By: Ó Cadhain, Máirtín" -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | --- Comment #5 from David Cook <dcook@prosentient.com.au> --- Warnings: This uses Zebra's scan feature to build up the list of authority records to use for updating, and this can be a resource intensive operation. It can use 100% CPU for the zebrasrv connection since it's doing a lot of rapid scanning. So just beware. This script will also only work as expected if your Zebra indexes are current. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- I suppose another way of doing this could've been to compile the list of authority IDs, grabbed all the bibs that use those authorities, inspect the headings to see if an update is needed per authority record... but I think that would've been a fairly cumbersome intensive process as well... This script is a blunt instrument but it's effective. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #7 from David Cook <dcook@prosentient.com.au> --- Oh also note that 100% is only for the scanning portion. For a Koha with 15000 authorities scanned, it took about 59 seconds. The actual updater uses say 30-50% of 1 CPU or less for the process. (The impact on the database is less obvious at a glance as I'm running external DB servers.) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Just double-checked the DB impact and there's pretty much zero impact. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- Ahh, I'm getting caught out by AuthorityMergeLimit... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #9)
Ahh, I'm getting caught out by AuthorityMergeLimit...
Although it looks like it put them in the need_merge_authorities table, so they will be processed eventually. That's probably actually a bit more optimal anyway. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|dcook@prosentient.com.au |koha-bugs@lists.koha-commun | |ity.org -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #11 from Nick Clemens <nick@bywatersolutions.com> --- This looks interesting - would be nice if adapted to work with ES as well -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Needs Signoff |Failed QA --- Comment #12 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- +1 for ES support I'm afraid. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|martin.renvoize@ptfs-europe | |.com | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #13 from Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> --- Is this something you might still work on David? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #14 from Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> --- https://stackoverflow.com/questions/14466274/query-all-unique-values-of-a-fi... looks promising on the ES front.. but I'm not sure how we'd achieve that in Koha land. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize (ashimema) from comment #13)
Is this something you might still work on David?
Mmm probably not. I can't recall who I did the Zebra version for, and I don't know that I've even thought about this again since 2021 😬 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28011 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|martin.renvoize@openfifth.c |testopia@bugs.koha-communit |o.uk |y.org -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org