https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |tomascohen@gmail.com |ity.org | Status|NEW |Needs Signoff CC| |tomascohen@gmail.com -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 --- Comment #2 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 122189 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=122189&action=edit Bug 28604: Prevent double encoding of MARC::Record::MiJ->to_mij output This patch fixes a double-encoding issue with MiJ output. Mojolicious' *text* renderer encodes the passed information according to the request context. [1] MARC::Record::MiJ->to_mij, conveniently encodes the string before output [2]. This causes double encoding. So the solution to this situation, is to use the *data* renderer, which doesn't perform any encoding [3]. To test: 1. Apply the regression tests patch 2. Run: $ kshell k$ prove t/db_dependent/api/v1/biblios.t => FAIL: Tests contain diacritics and fail! 3. Have a record with diacritics 4. Try the API routes for fetching a biblio: $ curl --location --request GET 'http://localhost:8080/api/v1/public/biblios/144' \ --header 'Accept: application/marc-in-json' (replace the record id with the one you've chosen) => FAIL: Boo, double encoding 5. Bonus point: you can try it on the non-public route, but you need more configuration boilerplate (basic auth, permissions). If you look at the fix, you will understand the tests cover it and no need to complicate yourself. 6. Apply this patch 7. Repeat 2 => SUCCESS: Tests pass! 8. Repeat 4 (and maybe 5) => SUCCESS: No double encoding! Yay! 9. Sign off :-D [1] https://metacpan.org/release/MRAMBERG/Convos-0.5/view/local/lib/perl5/Mojoli... [2] https://metacpan.org/dist/MARC-File-MiJ/source/lib/MARC/Record/MiJ.pm#L111 [3] https://metacpan.org/release/MRAMBERG/Convos-0.5/view/local/lib/perl5/Mojoli... Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #122189|0 |1 is obsolete| | --- Comment #4 from Lucas Gass <lucas@bywatersolutions.com> --- Created attachment 122191 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=122191&action=edit Bug 28604: Prevent double encoding of MARC::Record::MiJ->to_mij output This patch fixes a double-encoding issue with MiJ output. Mojolicious' *text* renderer encodes the passed information according to the request context. [1] MARC::Record::MiJ->to_mij, conveniently encodes the string before output [2]. This causes double encoding. So the solution to this situation, is to use the *data* renderer, which doesn't perform any encoding [3]. To test: 1. Apply the regression tests patch 2. Run: $ kshell k$ prove t/db_dependent/api/v1/biblios.t => FAIL: Tests contain diacritics and fail! 3. Have a record with diacritics 4. Try the API routes for fetching a biblio: $ curl --location --request GET 'http://localhost:8080/api/v1/public/biblios/144' \ --header 'Accept: application/marc-in-json' (replace the record id with the one you've chosen) => FAIL: Boo, double encoding 5. Bonus point: you can try it on the non-public route, but you need more configuration boilerplate (basic auth, permissions). If you look at the fix, you will understand the tests cover it and no need to complicate yourself. 6. Apply this patch 7. Repeat 2 => SUCCESS: Tests pass! 8. Repeat 4 (and maybe 5) => SUCCESS: No double encoding! Yay! 9. Sign off :-D [1] https://metacpan.org/release/MRAMBERG/Convos-0.5/view/local/lib/perl5/Mojoli... [2] https://metacpan.org/dist/MARC-File-MiJ/source/lib/MARC/Record/MiJ.pm#L111 [3] https://metacpan.org/release/MRAMBERG/Convos-0.5/view/local/lib/perl5/Mojoli... Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #122190|0 |1 is obsolete| | --- Comment #5 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 122292 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=122292&action=edit Bug 28604: Regression tests This patch introduces regression tests for the encoding issue with MiJ output. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |martin.renvoize@ptfs-europe | |.com --- Comment #7 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- All works well, QA script is happy, Tests pass. Passing QA -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 --- Comment #8 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Pushed to master for 21.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|21.11.00,21.05.02 |21.11.00,21.05.02,20.11.09 released in| | Status|Pushed to stable |Pushed to oldstable CC| |fridolin.somers@biblibre.co | |m --- Comment #10 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 20.11.x for 20.11.09 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 --- Comment #12 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Victor Grousset/tuxayo from comment #11)

Not backported to oldoldstable (20.05.x). Feel free to ask if it's needed.

Can you please backport it to the 20.05.x branch? Thanks :-) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wainuiwitikapark@catalyst.n | |et.nz --- Comment #15 from wainuiwitikapark@catalyst.net.nz --- Should this be backported to 19.11.x for 19.11.21? Or if it contains string changes, backported to 19.11.22? As we are currently in a string freeze. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 --- Comment #17 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to wainuiwitikapark from comment #16)

Should this be backported to 19.11.x for 19.11.22?

If it applies, backport it. It has no string changes -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 --- Comment #19 from wainuiwitikapark@catalyst.net.nz --- On 19.11.x Test Result (2 failures / ±0) t_db_dependent_api_v1_biblios_t.get() tests (703) t_db_dependent_api_v1_biblios_t.get_public() tests (703) I am getting these failtures Can someone please write a patch for that -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28604 --- Comment #21 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to wainuiwitikapark from comment #19)

On 19.11.x

Test Result (2 failures / ±0)

t_db_dependent_api_v1_biblios_t.get() tests (703) t_db_dependent_api_v1_biblios_t.get_public() tests (703)

I am getting these failtures

Can someone please write a patch for that

I tried to understand what's going on yesterday, and couldn't figure why it is failing, I agree with Victor, that this can safely be reverted to keep jenkins happy, and if someone requests it, we can come back to it. -- You are receiving this mail because: You are watching all bug changes.