Fundamental flaw in Koha REST API
Hi all, Just wondering if my email titled “x-koha-query header doesn't support UTF-8 data” actually made it to the koha-devel list as I didn’t get any replies to what I perceive as a significant issue. Basically, the x-koha-query header can only support ISO 8859-1/Latin-1 encoded characters. If you try to send UTF-8 encoded data to the API via Javascript, it will silently fail. See Bug 32401 for a practical example of this. I encountered this issue when using Arabic, but it is also obvious when using Hebrew, Chinese, Devanagari, or any non-Latin based script. It is less obvious – but still a problem – for some languages like Dutch, Finnish, French, and German. For instance, a x-koha-query header containing the French word “cœur” will fail, because œ is not a ISO 8859-1/Latin-1 character. (See the Wikipedia entry for ISO 8859-1 for more examples of problems with languages mostly covered by Latin-1.) Koha devs need to discuss this and possibly remove “x-koha-query” from Koha. At the moment, it’s not widely used by Koha itself, so I don’t think it will be hard to remove from Koha, but any third-party integrations would need to refactor to use a different option. Anyway, it would be great if someone could acknowledge receipt of this email. I have figured out how to get DataTables to stop using x-koha-query, so I’m going to do that for the particular acquisitions problem mentioned in Bug 32401, but we do need to talk about this issue. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595
Hi, On Mon, Dec 5, 2022 at 5:40 PM David Cook <dcook@prosentient.com.au> wrote:
At the moment, it’s not widely used by Koha itself, so I don’t think it will be hard to remove from Koha, but any third-party integrations would need to refactor to use a different option.
This might not be a huge factor, though of course removing that header should go through a deprecation procedure. Specifically, upon skimming the results of a GitHub search of "x-koha-query", the only uses I found outside of Koha itself were in plugins published by a couple active community members. Regards, Galen -- Galen Charlton Implementation and IT Manager Equinox Open Library Initiative gmc@equinoxOLI.org https://www.equinoxOLI.org phone: 877-OPEN-ILS (673-6457) direct: 770-709-5581
I recall we used the header at some point, to keep the base query clean when adding new filters. The only alternative is forcing the use of some form of ascii encoding (base64?) But I agree we should better deprecate it. It's use in the codebase on DT is tied to an option switch and probably not used El lun, 5 dic 2022 20:08, Galen Charlton <gmc@equinoxoli.org> escribió:
Hi,
On Mon, Dec 5, 2022 at 5:40 PM David Cook <dcook@prosentient.com.au> wrote:
At the moment, it’s not widely used by Koha itself, so I don’t think it will be hard to remove from Koha, but any third-party integrations would need to refactor to use a different option.
This might not be a huge factor, though of course removing that header should go through a deprecation procedure.
Specifically, upon skimming the results of a GitHub search of "x-koha-query", the only uses I found outside of Koha itself were in plugins published by a couple active community members.
Regards,
Galen -- Galen Charlton Implementation and IT Manager Equinox Open Library Initiative gmc@equinoxOLI.org https://www.equinoxOLI.org phone: 877-OPEN-ILS (673-6457) direct: 770-709-5581 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Perhaps Bug 30165 allowing multiple “q” parameters would provide an alternative as well? In the DataTables, it looks like it’s used for receiving acquisitions and something with POS registers. In master, the acquisitions issue is solved by turning off that option switch. (Although in older versions of Koha the Orders route doesn’t seem able to handle the UTF-8 encoded characters.) David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Tomas Cohen Arazi <tomascohen@gmail.com> Sent: Tuesday, 6 December 2022 11:24 AM To: Galen Charlton <gmc@equinoxoli.org> Cc: David Cook <dcook@prosentient.com.au>; koha-devel <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Fundamental flaw in Koha REST API I recall we used the header at some point, to keep the base query clean when adding new filters. The only alternative is forcing the use of some form of ascii encoding (base64?) But I agree we should better deprecate it. It's use in the codebase on DT is tied to an option switch and probably not used El lun, 5 dic 2022 20:08, Galen Charlton <gmc@equinoxoli.org <mailto:gmc@equinoxoli.org> > escribió: Hi, On Mon, Dec 5, 2022 at 5:40 PM David Cook <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > wrote:
At the moment, it’s not widely used by Koha itself, so I don’t think it will be hard to remove from Koha, but any third-party integrations would need to refactor to use a different option.
This might not be a huge factor, though of course removing that header should go through a deprecation procedure. Specifically, upon skimming the results of a GitHub search of "x-koha-query", the only uses I found outside of Koha itself were in plugins published by a couple active community members. Regards, Galen -- Galen Charlton Implementation and IT Manager Equinox Open Library Initiative gmc@equinoxOLI.org <mailto:gmc@equinoxOLI.org> https://www.equinoxOLI.org phone: 877-OPEN-ILS (673-6457) direct: 770-709-5581 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Btw looks like Bug 30165 is what resolves the UTF-8 issue for the orders route… David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: David Cook <dcook@prosentient.com.au> Sent: Tuesday, 6 December 2022 11:31 AM To: 'Tomas Cohen Arazi' <tomascohen@gmail.com>; 'Galen Charlton' <gmc@equinoxoli.org> Cc: 'koha-devel' <koha-devel@lists.koha-community.org> Subject: RE: [Koha-devel] Fundamental flaw in Koha REST API Perhaps Bug 30165 allowing multiple “q” parameters would provide an alternative as well? In the DataTables, it looks like it’s used for receiving acquisitions and something with POS registers. In master, the acquisitions issue is solved by turning off that option switch. (Although in older versions of Koha the Orders route doesn’t seem able to handle the UTF-8 encoded characters.) David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Tomas Cohen Arazi <tomascohen@gmail.com <mailto:tomascohen@gmail.com> > Sent: Tuesday, 6 December 2022 11:24 AM To: Galen Charlton <gmc@equinoxoli.org <mailto:gmc@equinoxoli.org> > Cc: David Cook <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> >; koha-devel <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: Re: [Koha-devel] Fundamental flaw in Koha REST API I recall we used the header at some point, to keep the base query clean when adding new filters. The only alternative is forcing the use of some form of ascii encoding (base64?) But I agree we should better deprecate it. It's use in the codebase on DT is tied to an option switch and probably not used El lun, 5 dic 2022 20:08, Galen Charlton <gmc@equinoxoli.org <mailto:gmc@equinoxoli.org> > escribió: Hi, On Mon, Dec 5, 2022 at 5:40 PM David Cook <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > wrote:
At the moment, it’s not widely used by Koha itself, so I don’t think it will be hard to remove from Koha, but any third-party integrations would need to refactor to use a different option.
This might not be a huge factor, though of course removing that header should go through a deprecation procedure. Specifically, upon skimming the results of a GitHub search of "x-koha-query", the only uses I found outside of Koha itself were in plugins published by a couple active community members. Regards, Galen -- Galen Charlton Implementation and IT Manager Equinox Open Library Initiative gmc@equinoxOLI.org <mailto:gmc@equinoxOLI.org> https://www.equinoxOLI.org phone: 877-OPEN-ILS (673-6457) direct: 770-709-5581 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
participants (3)
-
David Cook -
Galen Charlton -
Tomas Cohen Arazi