[Bug 11278] New: Adjusting bulkmarcimport.pl for customization routine and verbose printing
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Bug ID: 11278 Summary: Adjusting bulkmarcimport.pl for customization routine and verbose printing Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: ASSIGNED Severity: normal Priority: P5 - low Component: Command-line Utilities Assignee: m.de.rooy@rijksmuseum.nl Reporter: m.de.rooy@rijksmuseum.nl QA Contact: testopia@bugs.koha-community.org The forthcoming patch changes bulkmarcimport.pl in two ways: 1) For the verbose option, verbose level 2 now means print the formatted marcxml for the records. Verbose 1 just prints dots and counts-per-hundred-records. 2) The new code for customization allows you to add a LocalChanges.pm in the misc/migration_tools folder. If this module exists, it is 'required' and a routine customize is called for each record. This makes it possible to add local adjustments to your marc record before they are processed in bulkmarcimport. I will attach a second example patch with such a module just to show the idea; this patch does not need signoff, etc. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 --- Comment #1 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 23061 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23061&action=edit One test record in marcxml -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 --- Comment #2 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 23062 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23062&action=edit Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 --- Comment #3 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 23063 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23063&action=edit Bug 11278: [EXAMPLE: DO_NOT_PUSH] Example LocalChanges.pm This patch adds a LocalChanges.pm to migration_tools. To illustrate the concept of record customization during bulkmarcimport. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #23062|0 |1 is obsolete| | --- Comment #4 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 23133 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23133&action=edit Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 --- Comment #5 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 23134 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23134&action=edit Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA CC| |jonathan.druart@biblibre.co | |m --- Comment #6 from Jonathan Druart <jonathan.druart@biblibre.com> --- Hi Marcel, It does not seem to work with the default value for the custom parameter: % perl misc/migration_tools/bulkmarcimport.pl -file /tmp/zztest01.xml -v=2 -b -m XML -t | more TESTING MODE ONLY DOING NOTHING =============== Characteristic MARC flavour: UNIMARC LDR 00000uam a22000001i 4500 001 734 008 870805 041 _aeng 044 _cNL 070 _a86307796X 084 _a14 245 00 _aEmerging hierarchies : processes of stratification and early state formation in the Indonesian archipelago : prehistory and the ethnographic present _cdoor Ina Erna Slamet-Velsink 260 _a[S.l. _bs.n.] _c1986 300 _aviii, 310 p., 20 pl. _bill., maps _c22 cm 500 _aDoctoral diss. RU Leiden 500 _aBibliogr.: p. 278-296 500 _aIncludes index 650 4 _aarchaeology 650 4 _astates 650 4 _anation building 650 4 _acultural and social anthropology 651 4 _aindonesia 700 1 _aSlamet-Velsink, Ina Erna 852 _aCentrale Bibliotheek (CB) _cN 87-911 _p14000000001457 _zBeschikbaar 852 _aCentrale Bibliotheek (CB) _cN 96-317 _p14000002218786 _zBeschikbaar 1 MARC records done in 0.0246779918670654 seconds There is no 005 created. Marked as Failed QA. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #7 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #6)
Hi Marcel, It does not seem to work with the default value for the custom parameter:
% perl misc/migration_tools/bulkmarcimport.pl -file /tmp/zztest01.xml -v=2 -b -m XML -t | more
The default value will only be used if you supply the custom flag. Something like bulkmarcimport.pl -file ... -cust ... In your example the -custom parameter is not passed, so bulkmarcimport does not look at the customization routines (just normal behavior). Does this need a better explanation in the POD ? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #23133|0 |1 is obsolete| | --- Comment #8 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 24748 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24748&action=edit Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #23134|0 |1 is obsolete| | --- Comment #9 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 24749 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24749&action=edit Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 --- Comment #10 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 24750 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24750&action=edit Bug 11278: (follow-up) Return an exit value (1) if the module is not found. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 --- Comment #11 from Jonathan Druart <jonathan.druart@biblibre.com> --- (In reply to M. de Rooy from comment #7)
In your example the -custom parameter is not passed, so bulkmarcimport does not look at the customization routines (just normal behavior).
Yes sorry, I have just stupidly followed the test plan. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 --- Comment #12 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #11) Thanks for testing and follow-up. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de Severity|normal |enhancement -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #24748|0 |1 is obsolete| | --- Comment #13 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 26087 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26087&action=edit [PASSED QA] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #24749|0 |1 is obsolete| | --- Comment #14 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 26088 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26088&action=edit [PASSED QA] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #26088|0 |1 is obsolete| | --- Comment #15 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 26090 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26090&action=edit [PASSED QA] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #26087|0 |1 is obsolete| | --- Comment #16 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 26099 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26099&action=edit [PASSED QA] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #26090|0 |1 is obsolete| | --- Comment #17 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 26100 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26100&action=edit [PASSED QA] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #24750|0 |1 is obsolete| | --- Comment #18 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 26101 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26101&action=edit [PASSED QA] Bug 11278: (follow-up) Return an exit value (1) if the module is not found. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 --- Comment #19 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Does exactly what it says on the tin, and passes QA scripts too. I'm happy that the code all looks sane and reasonable, and have tested some larger imports with & without the given example filter module. I would love to see some of the bulkmarkimport features added to the stage_file.pl and commit_file.pl workflow for this action; but this is for another day. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_3_16_candidate -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11278 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master CC| |gmcharlt@gmail.com --- Comment #20 from Galen Charlton <gmcharlt@gmail.com> --- Pushed to master. Thanks, Marcel! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org