[Bug 18389] New: bulkmarcimport.pl does not take notice of MARC modification templates
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Bug ID: 18389 Summary: bulkmarcimport.pl does not take notice of MARC modification templates Change sponsored?: --- Product: Koha Version: 16.11 Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: MARC Bibliographic record staging/import Assignee: gmcharlt@gmail.com Reporter: J.P.Knight@lboro.ac.uk QA Contact: testopia@bugs.koha-community.org Created attachment 61907 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=61907&action=edit Patch for bulkmarcimport.pl When importing large numbers of MARC records from a legacy LMS to Koha we needed to convert some of 852 fields to 952 fields for item holdings. I'd created MARC modification templates in the web administration front end, but the bulkmarcimport.pl migration tool doesn't use these. Joubu on the #koha IRC channel told me that this was because bulkmarcimport.pl was written before C4::MarcModificationTemplates.pm Perl module was available. The change is actually relatively easy and I've attached a diff patch to this case for others to contemplate and critic. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Jon Knight <J.P.Knight@lboro.ac.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #61907|0 |1 is obsolete| | CC| |J.P.Knight@lboro.ac.uk --- Comment #1 from Jon Knight <J.P.Knight@lboro.ac.uk> --- Created attachment 63353 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63353&action=edit The same patch as above only generated within git and without touch existing whitespace formatting oddities in the script. When I mentioned this bug/patch on the Koha IRC channel originally I was told I needed to read up on how to submit bugs/patch because I'd done something wrong. I've read a load of wiki pages, etc now and I think the issue might have been that I tidied up the white space formatting in another part of the same script and didn't have the patch generated by git (rather I just used context diffing on our non-git installation). Hopefully this patch will find more favour. :-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Comment on attachment 63353 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63353 The same patch as above only generated within git and without touch existing whitespace formatting oddities in the script. Review of attachment 63353: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=18389&attachment=63353) ----------------------------------------------------------------- Hi Jon, the patch is looking good, thx for your contribution! Looking at the code, I am missing documentation of the new option - could you add it please so it shows up when bulkmarcimport is run without params or with -h? Also noticed another small thing: ::: misc/migration_tools/bulkmarcimport.pl @@ +3,4 @@
use strict; use warnings; +use Modern::Perl;
use Modern::Perl; replaces use strict; and use warnings; so the 2 lines above should be deleted. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de Status|NEW |Needs Signoff --- Comment #3 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Oh, and also don't forget to switch to "Needs sign-off" when you upload a patch for testing! Spotted this in my mailbox, but it will now show on the Dashboard etc. otherwise! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|16.11 |master Status|Needs Signoff |Failed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #4 from Jon Knight <J.P.Knight@lboro.ac.uk> --- Created attachment 63469 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63469&action=edit Additional patch to remove use strict/warn and add POD This is an additional patch to remove the use strict and use warn lines before the use Perl::Modern, and add some POD for the help option. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Jon Knight <J.P.Knight@lboro.ac.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #5 from Jon Knight <J.P.Knight@lboro.ac.uk> --- Oh, and thanks for the advice Katrin - I'm still getting to grips with Koha and git (been an RCS/SVN user for years!). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Thx for the follow-up! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA Assignee|gmcharlt@gmail.com |J.P.Knight@lboro.ac.uk CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #7 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Hi Jon, Could you resubmit your patch following the requirements on the wiki page https://wiki.koha-community.org/wiki/Commit_messages -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #8 from Jon Knight <J.P.Knight@lboro.ac.uk> --- OK. This may take some time as I'm new to git and I've not idea how to recommit something I've already committed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Jon Knight <J.P.Knight@lboro.ac.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #63353|0 |1 is obsolete| | Attachment #63469|0 |1 is obsolete| | --- Comment #9 from Jon Knight <J.P.Knight@lboro.ac.uk> --- Created attachment 63851 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=63851&action=edit Revised patch with commit information reformatted. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #10 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Sounds good, thanks Jon! Do not forget to change the status accordingly :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Jon, it would be great if this could also be included in the stage_file.pl/commit_file.pl scripts that are command line tools of the staged MARC import. Of course, separate bug (not trying to scope creep)! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- The QA tools point out some small issues with tabs vs. spaces - something to keep in mind for your next patches! As this works great, I am fixing them in the signed-off patch :) FAIL misc/migration_tools/bulkmarcimport.pl OK critic FAIL forbidden patterns forbidden pattern: tab char (line 126) forbidden pattern: tab char (line 128) forbidden pattern: tab char (line 285) forbidden pattern: tab char (line 125) forbidden pattern: tab char (line 131) forbidden pattern: tab char (line 127) forbidden pattern: tab char (line 286) OK git manipulation OK pod OK spelling OK valid -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #63851|0 |1 is obsolete| | --- Comment #13 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 66144 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=66144&action=edit [SIGNED OFF] Bug 18690: Use MARC modification templates with bulkmarcimport.pl When importing large numbers of MARC records from a legacy LMS to Koha using bulkmarcimport.pl, it did not make use of the MARC modification templates in the system (which can be useful for coversion of 852 fields to 952 fields for item holdings for example). This patch allows MARC modification templates to be used with bulkmarcimport.pl. To test: 1) Apply patch. 2) Set up a MARC modification template (in Home > Tools > MARC modification templates) to make some changes to imported MARC records (for example copy a subfield). 3) Take a test set of MARC records that have fields matching the template and import them using the bulkmarcimport.pl tool. For example if these MARC records are in testrecords.mrc and the MARC modification template is called testtemplate use something like: perl misc/migration_tools/bulkmarcimport.pl -commit 1000 \\ -file testrecords.mrc -marcmodtemplate testtemplate 4) Check the imported records in Koha to see that the required modifications have been applied when the MARC records are imported. 5) Sign off. https://bugs.koha-community.org/show_bug.cgi?id=18389 Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #14 from Jon Knight <J.P.Knight@lboro.ac.uk> --- Thanks Katrin! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|bulkmarcimport.pl does not |Allow using MARC |take notice of MARC |modification templates in |modification templates |bulkmarcimport.pl CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |tomascohen@gmail.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #15 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Hi Jon, the enhancement works as advertised and things look ok :-D I was wondering if using the template name was the best option, given the 'name' column is not a UNIQUE id. I saw the UI doesn't show the real DB ID for the MARC modification template, so I'm not sure what's the best solution. Please add your comments on how could we make it rock solid. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #16 from Jon Knight <J.P.Knight@lboro.ac.uk> --- I'd prefer to not expose the database ID to the end user as I don't think that's terribly user friendly, nor portable. Recently I've been writing Koha SQL reports for use with an external system and having the IDs change between our dev box and the (hosted) live Koha is a bit of a pain there (I'm tempted to suggest a bug enhancement to allow SQL reports to be selected by name rather than ID!). Should being able to generate MARC modification templates with identical names be viewed as a bug in the UI perhaps? Its going to be confusing there too. I'll put a check in to detect multiple MARC modification templates with the same name and refuse to run until they are renamed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #17 from Jon Knight <J.P.Knight@lboro.ac.uk> --- Created attachment 66379 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=66379&action=edit Added error if there are multiple MARC templates with the same name. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Jon Knight <J.P.Knight@lboro.ac.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #18 from Jon Knight <J.P.Knight@lboro.ac.uk> --- Test for this additional patch is to use the UI to create two MARC modification templates with exactly the same name, and then try to use that name with the bulkmarcimport.pl script. It should fail with an error message suggesting that the templates are renamed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #19 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Just a note: You can't actually rename the templates at the moment, but you can duplicate with a new name and delete the old one. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #66379|0 |1 is obsolete| | --- Comment #20 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 66498 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=66498&action=edit Added error if there are multiple MARC templates with the same name. https://bugs.koha-community.org/show_bug.cgi?id=18389 Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #66144|0 |1 is obsolete| | Attachment #66498|0 |1 is obsolete| | --- Comment #21 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 68587 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=68587&action=edit Bug 18690: Use MARC modification templates with bulkmarcimport.pl When importing large numbers of MARC records from a legacy LMS to Koha using bulkmarcimport.pl, it did not make use of the MARC modification templates in the system (which can be useful for coversion of 852 fields to 952 fields for item holdings for example). This patch allows MARC modification templates to be used with bulkmarcimport.pl. To test: 1) Apply patch. 2) Set up a MARC modification template (in Home > Tools > MARC modification templates) to make some changes to imported MARC records (for example copy a subfield). 3) Take a test set of MARC records that have fields matching the template and import them using the bulkmarcimport.pl tool. For example if these MARC records are in testrecords.mrc and the MARC modification template is called testtemplate use something like: perl misc/migration_tools/bulkmarcimport.pl -commit 1000 \\ -file testrecords.mrc -marcmodtemplate testtemplate 4) Check the imported records in Koha to see that the required modifications have been applied when the MARC records are imported. 5) Sign off. https://bugs.koha-community.org/show_bug.cgi?id=18389 Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #22 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 68588 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=68588&action=edit Added error if there are multiple MARC templates with the same name. https://bugs.koha-community.org/show_bug.cgi?id=18389 Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #23 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 68589 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=68589&action=edit Bug 18389: QA Followup - Fix tabs, note used template, info on verbose Fixed whitespace for QA tools Added a verbose note when template found Only print 'Modifying MARC' if verbose -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #68587|0 |1 is obsolete| | Attachment #68588|0 |1 is obsolete| | Attachment #68589|0 |1 is obsolete| | --- Comment #24 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 68590 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=68590&action=edit Bug 18690: Use MARC modification templates with bulkmarcimport.pl When importing large numbers of MARC records from a legacy LMS to Koha using bulkmarcimport.pl, it did not make use of the MARC modification templates in the system (which can be useful for coversion of 852 fields to 952 fields for item holdings for example). This patch allows MARC modification templates to be used with bulkmarcimport.pl. To test: 1) Apply patch. 2) Set up a MARC modification template (in Home > Tools > MARC modification templates) to make some changes to imported MARC records (for example copy a subfield). 3) Take a test set of MARC records that have fields matching the template and import them using the bulkmarcimport.pl tool. For example if these MARC records are in testrecords.mrc and the MARC modification template is called testtemplate use something like: perl misc/migration_tools/bulkmarcimport.pl -commit 1000 \\ -file testrecords.mrc -marcmodtemplate testtemplate 4) Check the imported records in Koha to see that the required modifications have been applied when the MARC records are imported. 5) Sign off. https://bugs.koha-community.org/show_bug.cgi?id=18389 Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #25 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 68591 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=68591&action=edit Added error if there are multiple MARC templates with the same name. https://bugs.koha-community.org/show_bug.cgi?id=18389 Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #26 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 68592 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=68592&action=edit Bug 18389: QA Followup - Fix tabs, note used template, info on verbose Fixed whitespace for QA tools Added a verbose note when template found Only print 'Modifying MARC' if verbose Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #27 from Nick Clemens <nick@bywatersolutions.com> --- Added a small followup, Tomas can you check it? Ideally we would fix the naming issue in the staff side, but I think is beyond scope here. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #28 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Nick Clemens from comment #27)
Added a small followup, Tomas can you check it?
Ideally we would fix the naming issue in the staff side, but I think is beyond scope here.
I like it, Nick. Go ahead. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #29 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 17.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #30 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Note: I did not catch before the wrong bug number in the commit message. It has been pushed with "bug 18690: ". Please do not be dyslexic with bug numbers. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #31 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Jonathan Druart from comment #30)
Note: I did not catch before the wrong bug number in the commit message. It has been pushed with "bug 18690: ". Please do not be dyslexic with bug numbers.
Will have to be manually fixed on release notes :( -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18389 --- Comment #32 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Ignore the sad smiley, just something to keep in mind. :) -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org