[Bug 30358] New: Strip leading/trailing whitespace characters from input fields when cataloguing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Bug ID: 30358 Summary: Strip leading/trailing whitespace characters from input fields when cataloguing Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Cataloging Assignee: koha-bugs@lists.koha-community.org Reporter: aleisha@catalyst.net.nz QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl Some libraries do a lot of copy pasting when cataloguing, which means occasionally copying over whitespace characters including line breaks and carriage breaks. It would be good if Koha stripped these. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Change sponsored?|--- |Sponsored Assignee|koha-bugs@lists.koha-commun |aleisha@catalyst.net.nz |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #1 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 132202 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132202&action=edit Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing WIP -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132202|0 |1 is obsolete| | --- Comment #2 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 132275 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132275&action=edit Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing This enhancement adds a system preference StripWhitespaceChars which, when enabled, will strip leading and trailing whitespace characters from all fields when cataloguing both bibliographic records and authority records. Whitespace characters that will be stripped are: - spaces - newlines - carriage returns - tabs To test: 1. Apply patch and install database updates 2. Go to Administration, system preferences, find the new StripWhitespaceChars preference. It should be "Don't strip" by default. Change it to "Strip". 3. Search for a biblio record and edit it. Put some leading or trailing whitespace characters in input fields and textarea fields and save. 4. Confirm these characters are removed when you save the record. 5. Repeat steps 3 and 4 for authority records. 6. Confirm tests pass t/db_dependent/Biblio/ModBiblioMarc.t Sponsored-by: Educational Services Australia SCIS -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=30443 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #3 from Aleisha Amohia <aleisha@catalyst.net.nz> --- *** Bug 30443 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #4 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Line breaks and carriage returns are known to break legacy library system record imports and, text area fields in MARC records are not designed to carry formatting, therefore it is implied that the data would be saved as a plain text string with no added formatting. For users who follow these cataloguing standards, there should be a setting to ask Koha to strip these. Planning to amend StripWhitespaceChars to also strip inner line breaks and carriage returns. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #5 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 132919 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132919&action=edit Bug 30358: (follow-up) Also strip inner newlines This patch amends the StripWhitespaceChars system preference to also strip inner newlines (line breaks and carriage returns) when enabled. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #6 from Owen Leonard <oleonard@myacpl.org> --- This seems to work but the test does pass: t/db_dependent/Biblio/ModBiblioMarc.t .. 1/2 # Failed test 'Title still has whitespace characters because StripWhitespaceChars is disabled' # at t/db_dependent/Biblio/ModBiblioMarc.t line 67. # got: 'My title' # expected: ' My title # ' # Looks like you failed 1 test of 3. # Failed test 'StripWhitespaceChars tests' # at t/db_dependent/Biblio/ModBiblioMarc.t line 75. # Looks like you failed 1 test of 2. t/db_dependent/Biblio/ModBiblioMarc.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/2 subtests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #7 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 133055 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133055&action=edit Bug 30358: (follow-up) Inner newlines should be replaced with a space -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #8 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 133056 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133056&action=edit Bug 30358: (follow-up) Fixing tests and including for inner newlines -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133056|0 |1 is obsolete| | --- Comment #9 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 133140 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133140&action=edit Bug 30358: (follow-up) Fixing tests and including for inner newlines -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to Aleisha Amohia from comment #4)
text area fields in MARC records are not designed to carry formatting, therefore it is implied that the data would be saved as a plain text string with no added formatting.
As far as I know, there is no prohibition against using carriage returns and line feeds in MARC records. The MARCXML schema seems to indicate that all whitespace is allowed: https://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd. That said, I haven't ever seen an example that includes them either.
For users who follow these cataloguing standards, there should be a setting to ask Koha to strip these.
Planning to amend StripWhitespaceChars to also strip inner line breaks and carriage returns.
It would probably be a good idea to indicate that it's data fields and not the leader and control fields that are affected. -- I'd also suggest adding an additional option for inner newlines. While I can see the utility of trimming whitespace off the beginning and end, I imagine that many librarians would be frustrated by inner whitespace being trimmed (at least if there wasn't a clear warning about it). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- As this is going to be tied to a new pref StripWhitespaceChars I think it will be ok. We had the same discussion I feel over adding HTML tags to records before. Ok, maybe that was a bit more extreme. I feel why some libraries won't want this, others will be happy as it helps when you copy&paste a lot of content from other sources. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #11)
As this is going to be tied to a new pref StripWhitespaceChars I think it will be ok. We had the same discussion I feel over adding HTML tags to records before. Ok, maybe that was a bit more extreme. I feel why some libraries won't want this, others will be happy as it helps when you copy&paste a lot of content from other sources.
Yeah, the system preference default looks fine. I think adding clear description in the system preference is a good idea though. The current wording is misleading. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133140|0 |1 is obsolete| | --- Comment #13 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 133224 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133224&action=edit Bug 30358: (follow-up) Fixing tests and including for inner newlines -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #14 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 133225 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133225&action=edit Bug 30358: (follow-up) Clarify syspref wording about fields affected -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #15 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 134195 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134195&action=edit Bug 30358: (follow-up) Consider field has multiple subfields of same key To test: 1) Click the clone subfield button to make multiple subfields with the same key, i.e. 500$a$a$a 2) Save the record and confirm that the fields contain the correct data after whitespaces are stripped. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #16 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 134453 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134453&action=edit Bug 30358: (follow-up) Put multiple subfields fix on auth side -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132275|0 |1 is obsolete| | --- Comment #17 from David Nind <david@davidnind.com> --- Created attachment 137451 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137451&action=edit Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing This enhancement adds a system preference StripWhitespaceChars which, when enabled, will strip leading and trailing whitespace characters from all fields when cataloguing both bibliographic records and authority records. Whitespace characters that will be stripped are: - spaces - newlines - carriage returns - tabs To test: 1. Apply patch and install database updates 2. Go to Administration, system preferences, find the new StripWhitespaceChars preference. It should be "Don't strip" by default. Change it to "Strip". 3. Search for a biblio record and edit it. Put some leading or trailing whitespace characters in input fields and textarea fields and save. 4. Confirm these characters are removed when you save the record. 5. Repeat steps 3 and 4 for authority records. 6. Confirm tests pass t/db_dependent/Biblio/ModBiblioMarc.t Sponsored-by: Educational Services Australia SCIS Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132919|0 |1 is obsolete| | --- Comment #18 from David Nind <david@davidnind.com> --- Created attachment 137452 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137452&action=edit Bug 30358: (follow-up) Also strip inner newlines This patch amends the StripWhitespaceChars system preference to also strip inner newlines (line breaks and carriage returns) when enabled. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133055|0 |1 is obsolete| | --- Comment #19 from David Nind <david@davidnind.com> --- Created attachment 137453 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137453&action=edit Bug 30358: (follow-up) Inner newlines should be replaced with a space Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133224|0 |1 is obsolete| | --- Comment #20 from David Nind <david@davidnind.com> --- Created attachment 137454 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137454&action=edit Bug 30358: (follow-up) Fixing tests and including for inner newlines Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133225|0 |1 is obsolete| | --- Comment #21 from David Nind <david@davidnind.com> --- Created attachment 137455 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137455&action=edit Bug 30358: (follow-up) Clarify syspref wording about fields affected Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #134195|0 |1 is obsolete| | --- Comment #22 from David Nind <david@davidnind.com> --- Created attachment 137456 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137456&action=edit Bug 30358: (follow-up) Consider field has multiple subfields of same key To test: 1) Click the clone subfield button to make multiple subfields with the same key, i.e. 500$a$a$a 2) Save the record and confirm that the fields contain the correct data after whitespaces are stripped. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #134453|0 |1 is obsolete| | --- Comment #23 from David Nind <david@davidnind.com> --- Created attachment 137457 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137457&action=edit Bug 30358: (follow-up) Put multiple subfields fix on auth side Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Medium patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA CC| |fridolin.somers@biblibre.co | |m --- Comment #24 from Fridolin Somers <fridolin.somers@biblibre.com> --- Looking for QA. Code added in C4/AuthoritiesMarc.pm and C4/Biblio.pm is exactly the same. Please move to a common method, maybe in Koha::Util::MARC. And add a unit test on this method. It will make a solid code ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #25 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 139110 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139110&action=edit Bug 30358: (follow-up) stripWhitespaceChars subroutine and tests To test: Confirm test plan above still works as expected and tests pass in t/Koha_MetadataRecord.t Sponsored-by: Catalyst IT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Patrick Robitaille <patrick.robitaille@collecto.ca> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |patrick.robitaille@collecto | |.ca -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Maude <maude.boudreau@collecto.ca> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |maude.boudreau@collecto.ca -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Status|Needs Signoff |Patch doesn't apply --- Comment #26 from David Nind <david@davidnind.com> --- Patch no longer applies... git bz apply 30358 Bug 30358 - Strip leading/trailing whitespace characters from input fields when cataloguing 137451 - Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing 137452 - Bug 30358: (follow-up) Also strip inner newlines 137453 - Bug 30358: (follow-up) Inner newlines should be replaced with a space 137454 - Bug 30358: (follow-up) Fixing tests and including for inner newlines 137455 - Bug 30358: (follow-up) Clarify syspref wording about fields affected 137456 - Bug 30358: (follow-up) Consider field has multiple subfields of same key 137457 - Bug 30358: (follow-up) Put multiple subfields fix on auth side 139110 - Bug 30358: (follow-up) stripWhitespaceChars subroutine and tests Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing .git/rebase-apply/patch:16: trailing whitespace. $record->add_fields($auth_type_tag,'','', $auth_type_subfield=>$authtypecode); warning: 1 line adds whitespace errors. Using index info to reconstruct a base tree... M C4/AuthoritiesMarc.pm M C4/Biblio.pm M installer/data/mysql/mandatory/sysprefs.sql M koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/cataloguing.pref M t/db_dependent/Biblio/ModBiblioMarc.t .git/rebase-apply/patch:16: trailing whitespace. $record->add_fields($auth_type_tag,'','', $auth_type_subfield=>$authtypecode); warning: 1 line adds whitespace errors. Falling back to patching base and 3-way merge... Auto-merging t/db_dependent/Biblio/ModBiblioMarc.t Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/cataloguing.pref CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/cataloguing.pref Auto-merging installer/data/mysql/mandatory/sysprefs.sql Auto-merging C4/Biblio.pm Auto-merging C4/AuthoritiesMarc.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #137451|0 |1 is obsolete| | --- Comment #27 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 142905 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142905&action=edit Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing This enhancement adds a system preference StripWhitespaceChars which, when enabled, will strip leading and trailing whitespace characters from all fields when cataloguing both bibliographic records and authority records. Whitespace characters that will be stripped are: - spaces - newlines - carriage returns - tabs To test: 1. Apply patch and install database updates 2. Go to Administration, system preferences, find the new StripWhitespaceChars preference. It should be "Don't strip" by default. Change it to "Strip". 3. Search for a biblio record and edit it. Put some leading or trailing whitespace characters in input fields and textarea fields and save. 4. Confirm these characters are removed when you save the record. 5. Repeat steps 3 and 4 for authority records. 6. Confirm tests pass t/db_dependent/Biblio/ModBiblioMarc.t Sponsored-by: Educational Services Australia SCIS Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #137452|0 |1 is obsolete| | --- Comment #28 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 142906 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142906&action=edit Bug 30358: (follow-up) Also strip inner newlines This patch amends the StripWhitespaceChars system preference to also strip inner newlines (line breaks and carriage returns) when enabled. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #137453|0 |1 is obsolete| | --- Comment #29 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 142907 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142907&action=edit Bug 30358: (follow-up) Inner newlines should be replaced with a space Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #137454|0 |1 is obsolete| | --- Comment #30 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 142908 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142908&action=edit Bug 30358: (follow-up) Fixing tests and including for inner newlines Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #137455|0 |1 is obsolete| | --- Comment #31 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 142909 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142909&action=edit Bug 30358: (follow-up) Clarify syspref wording about fields affected Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #137456|0 |1 is obsolete| | --- Comment #32 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 142910 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142910&action=edit Bug 30358: (follow-up) Consider field has multiple subfields of same key To test: 1) Click the clone subfield button to make multiple subfields with the same key, i.e. 500$a$a$a 2) Save the record and confirm that the fields contain the correct data after whitespaces are stripped. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #137457|0 |1 is obsolete| | --- Comment #33 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 142911 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142911&action=edit Bug 30358: (follow-up) Put multiple subfields fix on auth side Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139110|0 |1 is obsolete| | --- Comment #34 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 142912 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142912&action=edit Bug 30358: (follow-up) stripWhitespaceChars subroutine and tests To test: Confirm test plan above still works as expected and tests pass in t/Koha_MetadataRecord.t Sponsored-by: Catalyst IT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #35 from Fridolin Somers <fridolin.somers@biblibre.com> --- Note that line feed in title or author breaks items search because of Ajax via JSON -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #36 from David Nind <david@davidnind.com> --- Everything works as per the test plan, except for tests failing for t/db_dependent/Biblio/ModBiblioMarc.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #37 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 143053 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143053&action=edit Bug 30358: (follow-up) Fixing ModBiblioMarc.t tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142905|0 |1 is obsolete| | --- Comment #38 from David Nind <david@davidnind.com> --- Created attachment 143057 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143057&action=edit Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing This enhancement adds a system preference StripWhitespaceChars which, when enabled, will strip leading and trailing whitespace characters from all fields when cataloguing both bibliographic records and authority records. Whitespace characters that will be stripped are: - spaces - newlines - carriage returns - tabs To test: 1. Apply patch and install database updates 2. Go to Administration, system preferences, find the new StripWhitespaceChars preference. It should be "Don't strip" by default. Change it to "Strip". 3. Search for a biblio record and edit it. Put some leading or trailing whitespace characters in input fields and textarea fields and save. 4. Confirm these characters are removed when you save the record. 5. Repeat steps 3 and 4 for authority records. 6. Confirm tests pass t/db_dependent/Biblio/ModBiblioMarc.t Sponsored-by: Educational Services Australia SCIS Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142906|0 |1 is obsolete| | --- Comment #39 from David Nind <david@davidnind.com> --- Created attachment 143058 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143058&action=edit Bug 30358: (follow-up) Also strip inner newlines This patch amends the StripWhitespaceChars system preference to also strip inner newlines (line breaks and carriage returns) when enabled. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142907|0 |1 is obsolete| | --- Comment #40 from David Nind <david@davidnind.com> --- Created attachment 143059 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143059&action=edit Bug 30358: (follow-up) Inner newlines should be replaced with a space Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142908|0 |1 is obsolete| | --- Comment #41 from David Nind <david@davidnind.com> --- Created attachment 143060 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143060&action=edit Bug 30358: (follow-up) Fixing tests and including for inner newlines Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142909|0 |1 is obsolete| | --- Comment #42 from David Nind <david@davidnind.com> --- Created attachment 143061 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143061&action=edit Bug 30358: (follow-up) Clarify syspref wording about fields affected Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142910|0 |1 is obsolete| | --- Comment #43 from David Nind <david@davidnind.com> --- Created attachment 143062 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143062&action=edit Bug 30358: (follow-up) Consider field has multiple subfields of same key To test: 1) Click the clone subfield button to make multiple subfields with the same key, i.e. 500$a$a$a 2) Save the record and confirm that the fields contain the correct data after whitespaces are stripped. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142911|0 |1 is obsolete| | --- Comment #44 from David Nind <david@davidnind.com> --- Created attachment 143063 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143063&action=edit Bug 30358: (follow-up) Put multiple subfields fix on auth side Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142912|0 |1 is obsolete| | --- Comment #45 from David Nind <david@davidnind.com> --- Created attachment 143064 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143064&action=edit Bug 30358: (follow-up) stripWhitespaceChars subroutine and tests To test: Confirm test plan above still works as expected and tests pass in t/Koha_MetadataRecord.t Sponsored-by: Catalyst IT Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143053|0 |1 is obsolete| | --- Comment #46 from David Nind <david@davidnind.com> --- Created attachment 143065 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143065&action=edit Bug 30358: (follow-up) Fixing ModBiblioMarc.t tests Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #47 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Wondering if this is a bit too aggressively cleaning up? E.g. field 008 (and maybe more control fields) could end with a few spaces. You will remove them, which is not correct. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #48 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- About the regex: $value =~ s/^\s+|\s+$|^\n+|\n+$|^\r+|\r+$|^\t+|\t+$//g; Is the following not the same: $value =~ s/^\s+|\s+$//g; ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #49 from Aleisha Amohia <aleisha@catalyst.net.nz> --- (In reply to Marcel de Rooy from comment #47)
Wondering if this is a bit too aggressively cleaning up? E.g. field 008 (and maybe more control fields) could end with a few spaces. You will remove them, which is not correct.
Should I write a follow-up to exclude fields < 010? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #50 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Aleisha Amohia from comment #49)
(In reply to Marcel de Rooy from comment #47)
Wondering if this is a bit too aggressively cleaning up? E.g. field 008 (and maybe more control fields) could end with a few spaces. You will remove them, which is not correct.
Should I write a follow-up to exclude fields < 010?
Sounds reasonable to me. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143057|0 |1 is obsolete| | --- Comment #51 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144120 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144120&action=edit Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing This enhancement adds a system preference StripWhitespaceChars which, when enabled, will strip leading and trailing whitespace characters from all fields when cataloguing both bibliographic records and authority records. Whitespace characters that will be stripped are: - spaces - newlines - carriage returns - tabs To test: 1. Apply patch and install database updates 2. Go to Administration, system preferences, find the new StripWhitespaceChars preference. It should be "Don't strip" by default. Change it to "Strip". 3. Search for a biblio record and edit it. Put some leading or trailing whitespace characters in input fields and textarea fields and save. 4. Confirm these characters are removed when you save the record. 5. Repeat steps 3 and 4 for authority records. 6. Confirm tests pass t/db_dependent/Biblio/ModBiblioMarc.t Sponsored-by: Educational Services Australia SCIS Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143058|0 |1 is obsolete| | --- Comment #52 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144121 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144121&action=edit Bug 30358: (follow-up) Also strip inner newlines This patch amends the StripWhitespaceChars system preference to also strip inner newlines (line breaks and carriage returns) when enabled. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143059|0 |1 is obsolete| | --- Comment #53 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144122 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144122&action=edit Bug 30358: (follow-up) Inner newlines should be replaced with a space Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143060|0 |1 is obsolete| | --- Comment #54 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144123 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144123&action=edit Bug 30358: (follow-up) Fixing tests and including for inner newlines Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143061|0 |1 is obsolete| | --- Comment #55 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144124 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144124&action=edit Bug 30358: (follow-up) Clarify syspref wording about fields affected Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143062|0 |1 is obsolete| | --- Comment #56 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144125 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144125&action=edit Bug 30358: (follow-up) Consider field has multiple subfields of same key To test: 1) Click the clone subfield button to make multiple subfields with the same key, i.e. 500$a$a$a 2) Save the record and confirm that the fields contain the correct data after whitespaces are stripped. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143063|0 |1 is obsolete| | --- Comment #57 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144126 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144126&action=edit Bug 30358: (follow-up) Put multiple subfields fix on auth side Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143064|0 |1 is obsolete| | --- Comment #58 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144127 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144127&action=edit Bug 30358: (follow-up) stripWhitespaceChars subroutine and tests To test: Confirm test plan above still works as expected and tests pass in t/Koha_MetadataRecord.t Sponsored-by: Catalyst IT Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143065|0 |1 is obsolete| | --- Comment #59 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144128 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144128&action=edit Bug 30358: (follow-up) Fixing ModBiblioMarc.t tests Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #60 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144129 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144129&action=edit Bug 30358 (follow-up): Do not strip whitespace from control fields -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #61 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 144130 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144130&action=edit Bug 30358 (follow-up) Simplify regex The regex does the following: 1. Replace newlines and carriage returns with a space 2. Replace leading and trailing whitespace with nothing (strip) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #144130|0 |1 is obsolete| | --- Comment #62 from Hammat wele <hammat.wele@inlibro.com> --- Created attachment 144171 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144171&action=edit Bug 30358 (follow-up) Simplify regex The regex does the following: 1. Replace newlines and carriage returns with a space 2. Replace leading and trailing whitespace with nothing (strip) Signed-off-by: Hammat Wele <hammat.wele@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #144120|0 |1 is obsolete| | Attachment #144121|0 |1 is obsolete| | Attachment #144122|0 |1 is obsolete| | Attachment #144123|0 |1 is obsolete| | Attachment #144124|0 |1 is obsolete| | Attachment #144125|0 |1 is obsolete| | Attachment #144126|0 |1 is obsolete| | Attachment #144127|0 |1 is obsolete| | Attachment #144128|0 |1 is obsolete| | Attachment #144129|0 |1 is obsolete| | Attachment #144171|0 |1 is obsolete| | --- Comment #63 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145308 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145308&action=edit Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing This enhancement adds a system preference StripWhitespaceChars which, when enabled, will strip leading and trailing whitespace characters from all fields when cataloguing both bibliographic records and authority records. Whitespace characters that will be stripped are: - spaces - newlines - carriage returns - tabs To test: 1. Apply patch and install database updates 2. Go to Administration, system preferences, find the new StripWhitespaceChars preference. It should be "Don't strip" by default. Change it to "Strip". 3. Search for a biblio record and edit it. Put some leading or trailing whitespace characters in input fields and textarea fields and save. 4. Confirm these characters are removed when you save the record. 5. Repeat steps 3 and 4 for authority records. 6. Confirm tests pass t/db_dependent/Biblio/ModBiblioMarc.t Sponsored-by: Educational Services Australia SCIS Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #64 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145309 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145309&action=edit Bug 30358: (follow-up) Also strip inner newlines This patch amends the StripWhitespaceChars system preference to also strip inner newlines (line breaks and carriage returns) when enabled. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #65 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145310 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145310&action=edit Bug 30358: (follow-up) Inner newlines should be replaced with a space Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #66 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145311 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145311&action=edit Bug 30358: (follow-up) Fixing tests and including for inner newlines Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #67 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145312 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145312&action=edit Bug 30358: (follow-up) Clarify syspref wording about fields affected Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #68 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145313 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145313&action=edit Bug 30358: (follow-up) Consider field has multiple subfields of same key To test: 1) Click the clone subfield button to make multiple subfields with the same key, i.e. 500$a$a$a 2) Save the record and confirm that the fields contain the correct data after whitespaces are stripped. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #69 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145314 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145314&action=edit Bug 30358: (follow-up) Put multiple subfields fix on auth side Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #70 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145315 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145315&action=edit Bug 30358: (follow-up) stripWhitespaceChars subroutine and tests To test: Confirm test plan above still works as expected and tests pass in t/Koha_MetadataRecord.t Sponsored-by: Catalyst IT Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #71 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145316 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145316&action=edit Bug 30358: (follow-up) Fixing ModBiblioMarc.t tests Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #72 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145317 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145317&action=edit Bug 30358 (follow-up): Do not strip whitespace from control fields Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #73 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145318 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145318&action=edit Bug 30358 (follow-up) Simplify regex The regex does the following: 1. Replace newlines and carriage returns with a space 2. Replace leading and trailing whitespace with nothing (strip) Signed-off-by: Hammat Wele <hammat.wele@inlibro.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145308|0 |1 is obsolete| | Attachment #145317|0 |1 is obsolete| | Attachment #145318|0 |1 is obsolete| | --- Comment #74 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151202 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151202&action=edit Bug 30358: Strip leading/trailing whitespace characters from input fields when cataloguing This enhancement adds a system preference StripWhitespaceChars which, when enabled, will strip leading and trailing whitespace characters from all fields when cataloguing both bibliographic records and authority records. Whitespace characters that will be stripped are: - spaces - newlines - carriage returns - tabs To test: 1. Apply patch and install database updates 2. Go to Administration, system preferences, find the new StripWhitespaceChars preference. It should be "Don't strip" by default. Change it to "Strip". 3. Search for a biblio record and edit it. Put some leading or trailing whitespace characters in input fields and textarea fields and save. 4. Confirm these characters are removed when you save the record. 5. Repeat steps 3 and 4 for authority records. 6. Confirm tests pass t/db_dependent/Biblio/ModBiblioMarc.t Sponsored-by: Educational Services Australia SCIS Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145309|0 |1 is obsolete| | --- Comment #75 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151203 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151203&action=edit Bug 30358: (follow-up) Also strip inner newlines This patch amends the StripWhitespaceChars system preference to also strip inner newlines (line breaks and carriage returns) when enabled. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145310|0 |1 is obsolete| | --- Comment #76 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151204 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151204&action=edit Bug 30358: (follow-up) Inner newlines should be replaced with a space Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145311|0 |1 is obsolete| | --- Comment #77 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151205 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151205&action=edit Bug 30358: (follow-up) Fixing tests and including for inner newlines Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145312|0 |1 is obsolete| | --- Comment #78 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151206 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151206&action=edit Bug 30358: (follow-up) Clarify syspref wording about fields affected Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145313|0 |1 is obsolete| | --- Comment #79 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151207 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151207&action=edit Bug 30358: (follow-up) Consider field has multiple subfields of same key To test: 1) Click the clone subfield button to make multiple subfields with the same key, i.e. 500$a$a$a 2) Save the record and confirm that the fields contain the correct data after whitespaces are stripped. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145314|0 |1 is obsolete| | --- Comment #80 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151208 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151208&action=edit Bug 30358: (follow-up) Put multiple subfields fix on auth side Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145315|0 |1 is obsolete| | --- Comment #81 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151210 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151210&action=edit Bug 30358: (follow-up) stripWhitespaceChars subroutine and tests To test: Confirm test plan above still works as expected and tests pass in t/Koha_MetadataRecord.t Sponsored-by: Catalyst IT Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145316|0 |1 is obsolete| | --- Comment #82 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151212 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151212&action=edit Bug 30358: (follow-up) Fixing ModBiblioMarc.t tests Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #83 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151214 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151214&action=edit Bug 30358: (follow-up) Do not strip whitespace from control fields Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #84 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 151215 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151215&action=edit Bug 30358: (follow-up) Simplify regex The regex does the following: 1. Replace newlines and carriage returns with a space 2. Replace leading and trailing whitespace with nothing (strip) Signed-off-by: Hammat Wele <hammat.wele@inlibro.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #85 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- * Rebased, only conflict was in the .pref file * Fixed commit messages on the last 2 patches (QA script is happy now) * Ran the unit tests. All green * Verified controlfields are not affected, but other fields are cleaned up Back to Tomas :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #86 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Hi all. I'm willing to push a squashed version of this, because it's been hanging around for a while and it has an on/off switch and also (when squashed) seems like a small thing. I would've prefered more communication about this and a more QA eyes, because the chosen pattern is hard to accept in the current codebase (a static method on a not so related class). This should've been a Koha::RecordProcessor filter instead, as (a) This is not about Koha::Metadata::Record objects, but MARC::Record (b) that's what the Koha::RecordProcessor pattern is designed for. I will file a separate follow-up bug, and would like the authors to consider movign this code into a Koha::recordProcessor::Filter class. Happy to guide/help. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33749 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33749 [Bug 33749] Move Koha::MetadataRecord::stripWhitespaceChars into a RecordProcessor filter -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |23.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #87 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.05. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This adds a new system release notes| |preference | |`StripWhitespaceChars` | |which, when enabled, will | |strip leading and trailing | |whitespace characters from | |all fields when cataloguing | |both bibliographic records | |and authority records. | |Whitespace characters that | |will be stripped are: | | * | |spaces | | * newlines | | * | |carriage returns | | * tabs -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Documentation| |Caroline Cyr La Rose contact| | CC| |caroline.cyr-la-rose@inlibr | |o.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/692 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Pedro Amorim <pedro.amorim@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pedro.amorim@ptfs-europe.co | |m Status|Pushed to master |Needs documenting --- Comment #88 from Pedro Amorim <pedro.amorim@ptfs-europe.com> --- Enhancement. Not pushing to 22.11.x. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED --- Comment #89 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- Documentation already submitted -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #90 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Is the removal of inner newline chars on purpose? Doesn't it break note fields for some people -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358 --- Comment #91 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #90)
Is the removal of inner newline chars on purpose? Doesn't it break note fields for some people
Yeah I think it's on purpose, but I have the same criticism. I think it would break note fields for people. I raised some concerns back on https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30358#c10 I also suggested adding different options for trimming start/end and start/inner/end. I wouldn't recommend this system preference to anyone as is, but it should default to off, so caveat emptor... -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org