[Bug 41223] New: SUSHI harvest should respect rate limits
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Bug ID: 41223 Summary: SUSHI harvest should respect rate limits Initiative type: --- Sponsorship --- status: Product: Koha Version: 25.05 Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: ERM Assignee: jonathan.druart@gmail.com Reporter: jvinke@austinseminary.edu QA Contact: testopia@bugs.koha-community.org CC: jonathan.druart@gmail.com, jonathan.field@openfifth.co.uk, martin.renvoize@openfifth.co.uk, matt.blenkinsop@openfifth.co.uk, pedro.amorim@openfifth.co.uk Many SUSHI providers have rate limits that will prevent successful harvest of usage statistics if they are exceeded. If a harvest job results in a "Fatal - Client has made too many requests 1020" response, there is currently no recourse because rerunning the harvest will result in the issue. If there were either a built-in rate limit or a setting that could be applied to specific Data Providers it would resolve this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Mathieu Saby <mathsabypro@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathsabypro@gmail.com --- Comment #1 from Mathieu Saby <mathsabypro@gmail.com> --- +1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 --- Comment #2 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- KIT are actually working on this and I believe they are going to show us their work at Hackfest this year.. I'd like to help them upstream that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|jonathan.druart@gmail.com |martin.renvoize@openfifth.c | |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Hackfest -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |clemens.tubach@kit.edu, | |michaela.sieber@kit.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 --- Comment #3 from Michaela Sieber <michaela.sieber@kit.edu> --- Thanks Martin for adding us to CC. Yes, we have developed a plugin that retries background jobs which ended with error code 1011 or 1020. (see also SQL Report on Mattermost Channel ERM : https://chat.koha-community.org/koha-community/pl/egfbk37pjfgy3p6ppbznm99mfr ) Since we needed a quick solution, the plugin is currently compatible with the Koha version we are using in our production environment, which is 24.11.10 at the moment. It has not been tested with newer Koha versions yet, but I expect it should work with them as well. Our plugin is not yet publicly available, but I will inform you as soon as it is ready and published on GitLab. And yes, we will also present the plugin and a case study on COUNTER harvesting in Marseille this year, and probably at KohaCon as well. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=38804 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com --- Comment #4 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- What's the state of this bug post-Hackfest? Ideally we would solve this rate limit issue in core Koha rather than relying on a plugin. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|martin.renvoize@openfifth.c |matt.blenkinsop@openfifth.c |o.uk |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- I'm assigning this to Matt.. lets use the plugin as an example and submit a full fix to core. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 --- Comment #6 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Is the plugin now ready to be made public? If so I can take a look and work on fitting it into core -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41305 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41307 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 --- Comment #7 from Michaela Sieber <michaela.sieber@kit.edu> --- (In reply to Matt Blenkinsop from comment #6)
Is the plugin now ready to be made public? If so I can take a look and work on fitting it into core
https://gitlab.com/kit-library/koha-plugin-erm-harvesting-management -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|25.05 |Main Status|ASSIGNED |CONFIRMED --- Comment #8 from Michaela Sieber <michaela.sieber@kit.edu> --- The situation will improve somewhat with the planned ERM usage improvements, as we will then be able to identify more easily in the provider summary that a retry is required when COUNTER Error 1020 is returned. In addition, this could be handled automatically through retry management. Another approach could be to insert the background jobs created by the erm_run_harvester cron job into the database with a delay of 1–2 seconds between each job. Description Koha::ERM::EUsage::UsageDataProvider::enqueue_sushi_harvest_jobs enqueues the erm_sushi_harvester background jobs one right after the other, without any delay. As a result, the jobs for a provider [get identical or near-identical enqueued_on timestamps / hit the SUSHI service at the same moment . Proposed solution Add a one-second pause between consecutive enqueue calls in enqueue_sushi_harvest_jobs (no pause before the first or after the last job). Return value and parameters stay unchanged. A unit test should check the time difference between the enqueued_on values. Note The delay only affects enqueuing. If jobs should also be processed sequentially, that needs a different approach (e.g. a single worker or a dedicated queue). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41223 Lisa G <lgonzalez@palni.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lgonzalez@palni.edu -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org