[Bug 23873] New: Allow marc modification templates to use capturing groups in substitutions
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Bug ID: 23873 Summary: Allow marc modification templates to use capturing groups in substitutions Change sponsored?: --- Product: Koha Version: 19.05 Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Tools Assignee: koha-bugs@lists.koha-community.org Reporter: andrew@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org To test: - Find/create a bib record with a date formatted as YYYYMMDD - Create a marc modification template to Copy and Replace your field with this regex: s/(\d{4})(\d{2})(\d{2})/\1\-\2\-\3/ -- this should reformat your date as "YYYY-MM-DD" - Apply your template to your bib - Observe that the match worked, but the substitution failed -- the value "\1\-\2\-\3" is treated literally Apparently this fails due to some aspect of how regex works within perl? It'd be hugely useful if we could get it working. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joy@bywatersolutions.com, | |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- The issue here is that the two sides of the replacement are each passed in a variable. $value =~ s/$regex->{search}/$regex->{replace}/ On the right side, we don't evaluate the variable, so anything there is treated as text. This was encountered during the callnumber split development and is why the code there stores the full regex in a single field. https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variable... It seems maybe we can safely do this if we allow for /ee (double encoding) in the regex switch, then it will prevent malicious code, but allow desired code. In the case of callnumber splitting I think we decided too that we should trust the end user. Someone who already has access to the staff client is hopefully someone trustworthy -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org, | |julian.maurice@biblibre.com | |, katrin.fischer@bsz-bw.de, | |martin.renvoize@ptfs-europe | |.com, tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|major |enhancement Version|19.05 |master -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=21959 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 AspenCat Team <aspencatteam@clicweb.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aspencatteam@clicweb.org --- Comment #2 from AspenCat Team <aspencatteam@clicweb.org> --- I would also like to see substitution work in marc modification templates - or really anywhere we can use regex (such as batch item modification). This would really power-up the regex capabilities. Agree with Andrew that it would be hugely useful not just for date modification. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 --- Comment #3 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Are you thinking of working on this Nick.. it sounds like you had a hold of the issues it may present? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bernard.scaife@ptfs-europe. | |com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 --- Comment #4 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 128430 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128430&action=edit Bug 23873: Enable capture groups in MarcModificationTemplates This patch adds support for regular expression capture groups in MarcModificationTemplates. We escape double quotes in the replacement string, then quote the whole string before applying the `ee` modifier to the final regex application. See https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variable... for further details. Test plan 1) Attempt to use capture groups in your regular expression based MarcModificationTemplate and note it does not work. 2) Apply the patch and try again, this time the capture group should yield the expected results. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |martin.renvoize@ptfs-europe |ity.org |.com -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128430|0 |1 is obsolete| | --- Comment #5 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 128431 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128431&action=edit Bug 23873: Enable capture groups in MarcModificationTemplates This patch adds support for regular expression capture groups in MarcModificationTemplates. We escape double quotes in the replacement string, then quote the whole string before applying the `ee` modifier to the final regex application. See https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variable... for further details. Test plan 1) Attempt to use capture groups in your regular expression based MarcModificationTemplate and note it does not work. 2) Apply the patch and try again, this time the capture group should yield the expected results. 3) Run t/SimpleMARC.t and confirm the tests still pass. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 --- Comment #6 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 128432 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128432&action=edit Bug 23873: Unit tests This patch adds a unit test for capture groups support in Koha::SimpleMARC. Test plan 1) Run the test prior to applying the capture groups fix, note it fails. 2) Apply teh capture groups fix patch 3) Run the test again, it should now pass -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128431|0 |1 is obsolete| | --- Comment #7 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 128433 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128433&action=edit Bug 23873: Enable capture groups in MarcModificationTemplates This patch adds support for regular expression capture groups in MarcModificationTemplates. We escape double quotes in the replacement string, then quote the whole string before applying the `ee` modifier to the final regex application. See https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variable... for further details. Test plan 1) Attempt to use capture groups in your regular expression based MarcModificationTemplate and note it does not work. 2) Apply the patch and try again, this time the capture group should yield the expected results. 3) Run t/SimpleMARC.t and confirm the tests still pass. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 --- Comment #8 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Adding some details to the test plan, to spare others having to think about it :) 1 - have a record with 245a "the best book ever" 2 - create a modification template to copy and replace the 245a to the 245a, using regex 3 - in your regex, match "(.+\b)best(.+)" and replace with "$1worst$2" 4 - perform a batch mod on your record, end up with "$1worst$2" as your literal title 5) Apply the patch and try again, this time the capture group should yield the expected results. 6) Run t/SimpleMARC.t and confirm the tests still pass. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=29781 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128432|0 |1 is obsolete| | --- Comment #9 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 128980 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128980&action=edit Bug 23873: Unit tests This patch adds a unit test for capture groups support in Koha::SimpleMARC. Test plan 1) Run the test prior to applying the capture groups fix, note it fails. 2) Apply teh capture groups fix patch 3) Run the test again, it should now pass Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128433|0 |1 is obsolete| | --- Comment #10 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 128981 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128981&action=edit Bug 23873: Enable capture groups in MarcModificationTemplates This patch adds support for regular expression capture groups in MarcModificationTemplates. We escape double quotes in the replacement string, then quote the whole string before applying the `ee` modifier to the final regex application. See https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variable... for further details. Test plan 1) Attempt to use capture groups in your regular expression based MarcModificationTemplate and note it does not work. 2) Apply the patch and try again, this time the capture group should yield the expected results. 3) Run t/SimpleMARC.t and confirm the tests still pass. Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 --- Comment #11 from AspenCat Team <aspencatteam@clicweb.org> --- Since this works - YAY, could this be applied to batch item modification (such as changing a captured group in the call number and changing it to upper case)? I tried in sandbox after patch applied and did not work, should it have? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to AspenCat Team from comment #11)
Since this works - YAY, could this be applied to batch item modification (such as changing a captured group in the call number and changing it to upper case)? I tried in sandbox after patch applied and did not work, should it have?
Best to file a separate bug for this if there isn't one yet. They can be linked using 'see also'. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 --- Comment #13 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- (In reply to AspenCat Team from comment #11)
Since this works - YAY, could this be applied to batch item modification (such as changing a captured group in the call number and changing it to upper case)? I tried in sandbox after patch applied and did not work, should it have?
Good news! There'a already a separate bug for this. I just signed it off: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29781 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Manual -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128980|0 |1 is obsolete| | --- Comment #14 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 130759 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=130759&action=edit Bug 23873: Unit tests This patch adds a unit test for capture groups support in Koha::SimpleMARC. Test plan 1) Run the test prior to applying the capture groups fix, note it fails. 2) Apply teh capture groups fix patch 3) Run the test again, it should now pass Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128981|0 |1 is obsolete| | --- Comment #15 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 130760 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=130760&action=edit Bug 23873: Enable capture groups in MarcModificationTemplates This patch adds support for regular expression capture groups in MarcModificationTemplates. We escape double quotes in the replacement string, then quote the whole string before applying the `ee` modifier to the final regex application. See https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variable... for further details. Test plan 1) Attempt to use capture groups in your regular expression based MarcModificationTemplate and note it does not work. 2) Apply the patch and try again, this time the capture group should yield the expected results. 3) Run t/SimpleMARC.t and confirm the tests still pass. Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rajujogi.t@gmail.com --- Comment #16 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- *** Bug 29014 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |22.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 --- Comment #17 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to master for 22.05, thanks to everybody involved 🦄 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |40218 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=30233 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30233 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org