[Bug 32401] New: DataTable search can't handle UTF-8 when receiving orders
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Bug ID: 32401 Summary: DataTable search can't handle UTF-8 when receiving orders Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Acquisitions Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org If you try to search pending orders in acquisitions using Arabic like خمسة it will silently fail. Nothing changes on the screen. No "Processing" or obvious Javascript errors. However, if you turn on "Pause on caught exceptions", you'll see the following: TypeError: Failed to execute 'setRequestHeader' on 'XMLHttpRequest': String contains non ISO-8859-1 code point. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- Note that this issue exists in older versions of Koha too. The error is caught in the jQuery library, but I imagine the problem will be in the DataTables library. If you try it out at https://datatables.net/examples/api/multi_filter.html you'll see that there's no problem on the DataTables site, so I imagine it's either... a bug in our (old) version of DataTables, or a Koha bug... I'll take a bit more of a look to see if there's anything I can do... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- Ah, the problem is "x-koha-query". It looks like browsers don't like headers with non ISO-8859-1 values: "{"-and":[{"basket.vendor_id":46},[[{"basket.name":{"like":"%خمسة%"}}],[{"basket.basket_group.name":{"like":"%خمسة%"}}],[{"me.order_id":{"like":"%خمسة%"}}],[{"biblio.title":{"like":"%خمسة%"}},{"biblio.author":{"like":"%خمسة%"}},{"biblio.isbn":{"like":"%خمسة%"}},{"biblio.publisher":{"like":"%خمسة%"}},{"me.internal_note":{"like":"%خمسة%"}},{"me.vendor_note":{"like":"%خمسة%"}}],[{"me.replacement_price":{"like":"%خمسة%"}}],[{"me.quantity":{"like":"%خمسة%"}}],[{"me.ecost":{"like":"%خمسة%"}}],[{"fund.name":{"like":"%خمسة%"}}]]]}" According to https://www.rfc-editor.org/rfc/rfc7230#section-3.2.4 Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOULD limit their field values to US-ASCII octets. A recipient SHOULD treat other octets in field content (obs-text) as opaque data. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|major |critical Summary|DataTable search can't |x-koha-query cannot contain |handle UTF-8 when receiving |non-ISO-8859-1 values |orders | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32406 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com --- Comment #3 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Yes, we should remove it. I had a look at datatables.js and it seems safe to remove the related code there, we never pass query_parameters. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #3)
Yes, we should remove it.
I had a look at datatables.js and it seems safe to remove the related code there, we never pass query_parameters.
I think we need to push bug 32406 and do a similar fix for koha-tmpl/intranet-tmpl/prog/en/modules/pos/register.tt as they both rely on x-koha-query at the moment -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #3)
I had a look at datatables.js and it seems safe to remove the related code there, we never pass query_parameters.
Although we may not pass query_parameters, it seems that we create them within datatables.js when using header_filter: 669 if(options.header_filter) { 670 options.query_parameters = query_parameters; 671 } else { 672 dataSet.q = query_parameters; 673 delete options.query_parameters; 674 } -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32409 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 --- Comment #6 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 144450 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144450&action=edit Bug 32401: Remove x-koha-query support HTTP headers can only contain ASCII characters, and it is thus impractical to use a header for passing DB queries that could contain non-ascii characters. We don't need it, and should be removed. This patch removes traces from the datatables.js files and also from detail.tt, which had it set to false anyway. See bug 32406 and bug 32409 for the cases in which it was being used, and problematic. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed, | |rel_23_05_candidate -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #144450|0 |1 is obsolete| | --- Comment #7 from David Nind <david@davidnind.com> --- Created attachment 144462 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=144462&action=edit Bug 32401: Remove x-koha-query support HTTP headers can only contain ASCII characters, and it is thus impractical to use a header for passing DB queries that could contain non-ascii characters. We don't need it, and should be removed. This patch removes traces from the datatables.js files and also from detail.tt, which had it set to false anyway. See bug 32406 and bug 32409 for the cases in which it was being used, and problematic. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #8 from David Nind <david@davidnind.com> --- This is what I did to test - I hope this is sufficient (using KTD): 1. Repeat the steps in bug 32406 to set an order and then go to receive the order. 2. Enable the web developer tools for your browser on the page. 3. For Firefox: - select Network - change the requests that are displayed from "All" to "XHR" - refresh the page - click on the page in the results to display the headers - in the "Request Headers" section you should see: x-koha-query: {"basket.vendor_id":1} 4. After the patch is applied (and a restart_all) the x-koha-query is no longer displayed. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl --- Comment #9 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to David Cook from comment #0)
If you try to search pending orders in acquisitions using Arabic like خمسة it will silently fail. Nothing changes on the screen. No "Processing" or obvious Javascript errors.
However, if you turn on "Pause on caught exceptions", you'll see the following:
TypeError: Failed to execute 'setRequestHeader' on 'XMLHttpRequest': String contains non ISO-8859-1 code point.
Just a dumb question, but since Koha is multilingual and filtering on UTF8 chars does not really sound asking too much ? Why dont we use encodeURIComponent on the header string and call url_decode_utf8 on the receiving side? This could be kind of a convention for x-koha headers? Search for percent encoding, evaluate results of decoding? And be done quite early in the process on a general level when processing (PSGI) headers? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to Marcel de Rooy from comment #9)
Why dont we use encodeURIComponent on the header string and call url_decode_utf8 on the receiving side?
This could be kind of a convention for x-koha headers? Search for percent encoding, evaluate results of decoding? And be done quite early in the process on a general level when processing (PSGI) headers?
It sounds hacky to me. I've never heard of any other system doing something like that. Technically we could do it, but I think we should ask ourselves why we'd want to. (You make an interesting point about that on bug 32406 of course.) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |tomascohen@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 --- Comment #11 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Just for reference:
From RFC 8187 (Proposed standard) Indicating Character Encoding and Language for HTTP Header Field Parameters Extended notation, using the Unicode character U+00A3 ("£", POUND SIGN): foo: bar; title*=utf-8'en'%C2%A3%20rates Note: The Unicode pound sign character U+00A3 was encoded into the octet sequence C2 A3 using the UTF-8 character encoding, and then percent-encoded. Also, note that the space character was encoded as %20, as it is not contained in attr-char.
-- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |32406, 32409 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32406 [Bug 32406] Cannot search pending orders using non-latin-1 scripts https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32409 [Bug 32409] Cannot search cashups using non-latin-1 scripts -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|32406, 32409 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32406 [Bug 32406] Cannot search pending orders using non-latin-1 scripts https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32409 [Bug 32409] Cannot search cashups using non-latin-1 scripts -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |32406, 32409 See Also|https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32406, | |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32409 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32406 [Bug 32406] Cannot search pending orders using non-latin-1 scripts https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32409 [Bug 32409] Cannot search cashups using non-latin-1 scripts -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #144462|0 |1 is obsolete| | --- Comment #12 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 145926 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145926&action=edit Bug 32401: Remove x-koha-query support HTTP headers can only contain ASCII characters, and it is thus impractical to use a header for passing DB queries that could contain non-ascii characters. We don't need it, and should be removed. This patch removes traces from the datatables.js files and also from detail.tt, which had it set to false anyway. See bug 32406 and bug 32409 for the cases in which it was being used, and problematic. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |23.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 --- Comment #13 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.05. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Jacob O'Mara <jacob.omara@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |Pushed to stable Version(s)|23.05.00 |23.05.00,22.11.03 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 --- Comment #14 from Jacob O'Mara <jacob.omara@ptfs-europe.com> --- Nice work, thanks everyone! Pushed to 22.11.x for the next release. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com --- Comment #15 from Lucas Gass <lucas@bywatersolutions.com> --- Doesn't apply cleanly to 22.05.x, if needed please rebase. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|rel_23_05_candidate | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33971 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33971 [Bug 33971] Remove support for x-koha-query header -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32401 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |38243 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38243 [Bug 38243] DT's header_filter is unused -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org