[Bug 25952] New: Github search for plugin throwing errors for % and koha-plugin search terms
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Bug ID: 25952 Summary: Github search for plugin throwing errors for % and koha-plugin search terms Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: minor Priority: P5 - low Component: Plugin architecture Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org I was trying to get a full list of the available plugins so I tried '%' and 'koha-plugin' as search terms, but they created errors in Koha. Note that the 'koha-plugin' query works directly: https://api.github.com/search/repositories?q=koha-plugin+user:thekesolutions... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- It looks like it's due to rate limiting. I can use the search API: https://api.github.com/search/repositories?q=koha-plugin+user:bywatersolutio... But I can't get the releases: https://api.github.com/repos/bywatersolutions/koha-plugin-coverflow/releases... I guess that's because I've done < 60 searches, but each search has to do X release API calls, so it would be easy to exceed 60 API calls quickly if a person was looking for "koha-plugin" which would retrieve everything. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- Created attachment 106650 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=106650&action=edit Bug 25952: Add error-handling to prevent errors from rate limiting This patch makes sure there is a successful response before trying to decode the response. If you search for "koha-plugin" in the Bywater and Theke repos, you'll quickly exceed your API rate limit, and you'll stop getting successful responses. To test: 0) Don't apply patch yet 1) Enable plugins 2) Enable plugin_repos 3) Search for koha-plugin a few times 4) Notice how Plack crashes with a visible error in the browser 5) Apply the patch 6) restart_all (in koha-testing-docker) 7) Search for koha-plugin 8) Observe that no search results are returned and there are no errors in the browser -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |23975 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23975 [Bug 23975] Add ability to search and install plugins from GitHub -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Github search for plugin |Github search for plugin |throwing errors for % and |throwing errors due to API |koha-plugin search terms |rate limits -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|23975 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23975 [Bug 23975] Add ability to search and install plugins from GitHub -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |magnus@libriotech.no --- Comment #3 from Magnus Enger <magnus@libriotech.no> --- I enabled the feature by adding this to koha-conf.xml: <plugin_repos> <repo> <name>ByWater Solutions</name> <org_name>bywatersolutions</org_name> <service>github</service> </repo> <repo> <name>Theke Solutions</name> <org_name>thekesolutions</org_name> <service>github</service> </repo> </plugin_repos> Without the patch I always get this error: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/JSON.pm line 171 Is this the expected error we are trying to fix here? After applying the patch I searched for "koha-plugin" and got a bunch of hits, but only from Bywater. When I did the search again straight away it chewed for a while, then I got nothing. That is better than an ugly error message, of course, but wouldn't a friendly message like "oops something went wrong, please adjust your search or try again later" be better? Perhaps we could also intercept a search for "koha-plugin" and warn that it might exceed the rate limits and give unexepected results? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- (In reply to Magnus Enger from comment #3)
I enabled the feature by adding this to koha-conf.xml:
<plugin_repos> <repo> <name>ByWater Solutions</name> <org_name>bywatersolutions</org_name> <service>github</service> </repo> <repo> <name>Theke Solutions</name> <org_name>thekesolutions</org_name> <service>github</service> </repo> </plugin_repos>
Without the patch I always get this error:
malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/JSON.pm line 171
Is this the expected error we are trying to fix here?
It's been a while but I think so.
After applying the patch I searched for "koha-plugin" and got a bunch of hits, but only from Bywater. When I did the search again straight away it chewed for a while, then I got nothing. That is better than an ugly error message, of course, but wouldn't a friendly message like "oops something went wrong, please adjust your search or try again later" be better?
Perhaps we could also intercept a search for "koha-plugin" and warn that it might exceed the rate limits and give unexepected results?
That's a good point, although at this point I think I'm abandoning/shelving my work on this, as I think using the GitHub/GitLab APIs is a flawed approach overall. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion --- Comment #5 from Magnus Enger <magnus@libriotech.no> --- It does seem a bit brittle if we are already reaching the limits of the API. Do you have another approach in mind or do you think the whole “search and install” feature should be reverted? Setting to “in discussion”. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |23975 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23975 [Bug 23975] Add ability to search and install plugins from GitHub -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- (In reply to Magnus Enger from comment #5)
It does seem a bit brittle if we are already reaching the limits of the API. Do you have another approach in mind or do you think the whole “search and install” feature should be reverted?
I don't think that it should be reverted, but I think a manifest based approach would be more robust. It's something I plan to work on eventually one day in the future. Basically, we'd point the Koha config at a set of URLs (which are either for files or directories depending on the implementation) and we'd download manifests of available Koha plugins. (Basically, it's the same model as package managers.) The neat part of that would also be that sysadmins could choose which manifests to point at or even point to a local one of curated plugins. It would be quite flexible. The only downside I see with the manifests is that they would need to be updated, but I think that's really more of a plugin developer issue than a Koha issue. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #7 from Magnus Enger <magnus@libriotech.no> --- (In reply to David Cook from comment #6)
Basically, we'd point the Koha config at a set of URLs (which are either for files or directories depending on the implementation) and we'd download manifests of available Koha plugins. (Basically, it's the same model as package managers.)
Ah, sounds good to me! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Bug 25952 depends on bug 23975, which changed state. Bug 23975 Summary: Add ability to search and install plugins from GitHub https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23975 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|dcook@prosentient.com.au |koha-bugs@lists.koha-commun | |ity.org -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Bug 25952 depends on bug 23975, which changed state. Bug 23975 Summary: Add ability to search and install plugins from GitHub https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23975 What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED |--- -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Bug 25952 depends on bug 23975, which changed state. Bug 23975 Summary: Add ability to search and install plugins from GitHub https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23975 What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #8 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 189584 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189584&action=edit Bug 25952: Skip invalid repositories when searching for plugins This patch prevents the end-user to get the ugly 500 "malformed JSON string" when something wrong happens with the plugin search. It happens currently in ktd with the plugin repos listed in $KOHA_CONF An error is reported to the UI -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #9 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 189585 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189585&action=edit Bug 25952: Skip invalid repositories when searching for plugins This patch prevents the end-user to get the ugly 500 "malformed JSON string" when something wrong happens with the plugin search. It happens currently in ktd with the plugin repos listed in $KOHA_CONF An error is reported to the UI -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com Attachment #189584|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff --- Comment #10 from Jonathan Druart <jonathan.druart@gmail.com> --- This patch fixes a plugin that appears in ktd recently (we added a list of plugin repos, and ptfs-e's one is wrong). I think it's ready for SO even if not perfect. It replaces David's patch (I hadn't seen it before). Feel free to obsolete if you disagree. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #106650|0 |1 is obsolete| | Attachment #189585|0 |1 is obsolete| | --- Comment #11 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 189597 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189597&action=edit Bug 25952: Skip invalid repositories when searching for plugins This patch prevents the end-user to get the ugly 500 "malformed JSON string" when something wrong happens with the plugin search. It happens currently in ktd with the plugin repos listed in $KOHA_CONF An error is reported to the UI Signed-off-by: Owen Leonard <oleonard@myacpl.org> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bibliothek@th-wildau.de --- Comment #12 from Jan Kissig <bibliothek@th-wildau.de> --- With the patch applied: I just ran ktd with the --plugins option which enables the following repos: <plugin_repos> <repo> <name>ByWater Solutions</name> <org_name>bywatersolutions</org_name> <service>github</service> </repo> <repo> <name>Theke Solutions</name> <org_name>thekesolutions</org_name> <service>github</service> </repo> <repo> <name>PTFS Europe</name> <org_name>PTFS-Europe</org_name> <service>github</service> </repo> <repo> <name>Solutions inLibro</name> <org_name>inLibro</org_name> <service>github</service> </repo> </plugin_repos> Then I searched for koha-plugin plugins-home.pl and it only returned a result for PTFS Cannot search repository PTFS Europe: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/JSON.pm line 190. I did not get any other error for the other repos. This just calls for the proposed plugin store Open Fifth presented :) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41032 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #13 from Magnus Enger <magnus@libriotech.no> --- (In reply to Jan Kissig from comment #12)
Then I searched for koha-plugin plugins-home.pl and it only returned a result for PTFS
Cannot search repository PTFS Europe: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/JSON.pm line 190. I did not get any other error for the other repos.
That error should go away if you delete or update the config for the PTFS Europe repo, ref bug 41032. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #14 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 189848 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189848&action=edit Bug 25952: Skip invalid repositories when searching for plugins This patch prevents the end-user to get the ugly 500 "malformed JSON string" when something wrong happens with the plugin search. It happens currently in ktd with the plugin repos listed in $KOHA_CONF An error is reported to the UI Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #189597|0 |1 is obsolete| | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #15 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Tested using the default (wrong) config provided by KTD, which still ships with `PTFS-Europe`. The result is the expected: 1. Search for 'rapido' on the plugins page => SUCCESS: A warning about an error searching PTFS Europe => SUCCESS: The `koha-plugin-rapido-ill` plugin is displayed and can be installed -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Github search for plugin |Github search errors make |throwing errors due to API |it impossible to install |rate limits |plugins from other repos QA Contact|testopia@bugs.koha-communit |tomascohen@gmail.com |y.org | Severity|minor |normal Assignee|koha-bugs@lists.koha-commun |jonathan.druart@gmail.com |ity.org | CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |25.11.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #16 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Nice work everyone! Pushed to main for 25.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Text to go in the| |This fixes the error 500 release notes| |"malformed JSON string" | |message when something goes | |wrong searching for plugins | |using the plugin search in | |the staff interface (for | |example, when there is an | |invalid repository in the | |Koha configuration). Version(s)|25.11.00 | released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |25.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Paul Derscheid <paul.derscheid@lmscloud.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Pushed to stable Version(s)|25.11.00 |25.11.00,25.05.06 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #17 from Paul Derscheid <paul.derscheid@lmscloud.de> --- Nice work everyone! Pushed to 25.05.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Baptiste Wojtkowski (bwoj) <baptiste.wojtkowski@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|25.11.00,25.05.06 |25.11.00,25.05.06,24.11.12 released in| | CC| |baptiste.wojtkowski@biblibr | |e.com --- Comment #18 from Baptiste Wojtkowski (bwoj) <baptiste.wojtkowski@biblibre.com> --- Pushed into 24.11.x for 24.11.11 nice work everyone -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 --- Comment #19 from Baptiste Wojtkowski (bwoj) <baptiste.wojtkowski@biblibre.com> --- Pushed into 24.11.x for 24.11.12 nice work everyone -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Baptiste Wojtkowski (bwoj) <baptiste.wojtkowski@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldstable |Needs documenting CC| |wainuiwitikapark@catalyst.n | |et.nz --- Comment #20 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Not backporting to 22.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25952 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED --- Comment #21 from David Nind <david@davidnind.com> --- Bug fix, no change to the manual required. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org