[Bug 40394] New: Coce cover images should be requested in chunks
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40394 Bug ID: 40394 Summary: Coce cover images should be requested in chunks Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Templates Assignee: oleonard@myacpl.org Reporter: tomascohen@gmail.com QA Contact: testopia@bugs.koha-community.org If you choose to display 100 results per page when searching the catalog, the current Coce client implementation will pick all the identifiers, and make an API call to Coce for the 100 ids. This is problematic because: * It can be overwhelming for the Coce server (it will make a thread/worker busy for too long even when another is idle). * It makes the rendering slow or unreliable. We should at least split the ids list into chunks of some fixed (limited) size. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40394 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |f.demians@tamil.fr, | |lucas@bywatersolutions.com, | |oleonard@myacpl.org, | |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40394 --- Comment #1 from Frédéric Demians <f.demians@tamil.fr> --- A Coce server will lag heavily, and even will dysfunction, if you send too many ISBN... But how many is too many. It depends of the provider (aws,gb,ol). They way Amazon covers are retrieved (sequentially and with spacing) implies that for a large number of ISBN the global Coce timeout will be reached => cascading issues. Max 50 is OK. Above 100 is problematic. For OL, the response time is proportional to the number if ISBN, and so above 100, it take more than let say 7-9s... For GB, ISBN are truncate to 80-100, I don't know. Here an example of a Coce JS client, sending queries for batch of max 100 ISBN: https://github.com/fredericd/coce-rs/blob/master/client-sample/coceclient.js This considerations applies also to other Koha covers retrieving methods... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40394 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|oleonard@myacpl.org |koha-bugs@lists.koha-commun | |ity.org -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40394 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |tomascohen@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40394 Frédéric Demians <f.demians@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40394 --- Comment #2 from Frédéric Demians <f.demians@tamil.fr> --- Created attachment 207684 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=207684&action=edit Bug 40394: Request Coce cover images in chunks The Coce client sent every identifier found on the page in a single request. With many results per page, this overloads the Coce server, and a Coce server limiting the number of IDs per request (100 by default) rejects it: no cover is displayed at all. This patch replaces the JSONP (very old school technique) call with a small Coce client that sends the identifiers in chunks of at most 100, in parallel. It also caches the answers for the life of the page. The previous code removed 1x1 pixel images (Amazon placeholders) once loaded. This is no longer needed. Coce filters them out at the source. As the client now uses fetch() instead of JSONP, a Content-Security-Policy must allow the Coce server in connect-src (it was script-src before). Some tweaking may be necessary due to recent Koha CSP controls. TEST PLAN / KTD instance / default DB / MARC21 : 1. Set the sysorefs : - OpacCoce: Enable - CoceHost: https://coce.tamil.fr - CoceProviders: all - OPACnumSearchResults: 200 2. In the OPAC, search for "the" (more than 100 results with an ISBN on the page). Open the browser developer tools, Network tab, filter on "coce.tamil.fr". - A single request is sent to Coce, with all the ISBNs. - Coce answers 400 "Too many IDs, maximum is 100". - No cover image from Coce is displayed. 3. Apply the patch, reload the page (with cache reset) => Two requests are sent to Coce: one with 100 ISBNs, one with the remaining ones. Both answer 200. => Cover images are displayed. 4. Set OPACnumSearchResults back to 20, search again. => A single request, cover images displayed. 5. Open the detail page of a record with an ISBN. => Its cover image is displayed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40394 --- Comment #3 from Frédéric Demians <f.demians@tamil.fr> --- Created attachment 207685 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=207685&action=edit Bug 40394: Keep Coce cover images when Coce answers slowly Related to previous patch. If OPAC result page display a lot of biblio record, more than 100, Code will take time to responds. Let see how... The OPAC waits at most 5 seconds for cover images After this delay, it verifies the cover containers and removes those without an image. When Coce answers later, for example for covers it doesn't have in its cache yet, containers are already gone => no Coce cover is displayed. This patch verifies a container waiting for Coce once Coce has answered and the images it gave are loaded, instead of removing it. Other containers are verified as before. What's new in KOHA.coce : - loaded: a promise settled when all getURL() calls are finished - isPending(container): whether a Coce thumbnail is still waiting - whenLoaded(container): settles when Coce has answered and the container's Coce images are loaded verify_cover_images() verifies each container with a new verify_cover_slider() function. TEST PLAN / default KTD instance / MARC21 : 0. Set OPACnumSearchResults to 200. 1. Enable Coce: - OpacCoce: Enable - CoceProviders: check Amazon, Google Books and Open Library - CoceHost: https://coce.tamil.fr/delay/8000 (Coce answers after 8 seconds, longer than the 5 seconds the OPAC waits for cover images) 2. In the OPAC, search for "the" (it returns records with an ISBN). 3. Wait 10 seconds => No cover image from Coce is displayed. 4. Open the detail page of a record with an ISBN, e.g. /cgi-bin/koha/opac-detail.pl?biblionumber=70, and wait 10 seconds. => No cover image is displayed. 5. Apply the patch, then reload both pages (force browser cache reste to get new coce.js) 6. Repeat 2. Wait 10 seconds on each page. => Covers are displayed on the results page and the detail page, about 8 seconds after the page has loaded. 7. Set CoceHost back to https://coce.tamil.fr and reload both pages. => Covers are displayed right away, as before the patch. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org