[Bug 41294] New: Data not available for certain months should not result in a complete failure of the harvest
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Bug ID: 41294 Summary: Data not available for certain months should not result in a complete failure of the harvest Initiative type: --- Sponsorship --- status: Product: Koha Version: 24.11 Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: ERM Assignee: jonathan.druart@gmail.com Reporter: mathsabypro@gmail.com QA Contact: testopia@bugs.koha-community.org CC: jonathan.druart@gmail.com, jonathan.field@openfifth.co.uk, martin.renvoize@openfifth.co.uk, matt.blenkinsop@openfifth.co.uk, pedro.amorim@openfifth.co.uk I don't know if it's related to the SUSHI protocol itself (in that case there is nothing to do...) or to Koha, but it's annoying. Ex : for a data source if you try to harvest march and april 2025, and april data are not available, the harvest will fail with this error : Sushi harvesting return an error if some months are not available : Usage Not Ready for Requested Dates - Usage not yet released for the following yearmonths(s): 202504 3031 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew.auld@openfifth.co.uk | |, clemens.tubach@kit.edu, | |michaela.sieber@kit.edu --- Comment #1 from Michaela Sieber <michaela.sieber@kit.edu> --- Hm yes, that's why we only do monthly harvesting. It could be an option to force the monthly harvesting in the background and start separate background-jobs per month, even if users enter a period with several months -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|24.11 |Main Assignee|jonathan.druart@gmail.com |koha-bugs@lists.koha-commun | |ity.org -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Comma delimited| |Karlsruhe Institute of list of Sponsors| |Technology (KIT) Priority|P5 - low |P2 Blocks| |43904 Status|NEW |ASSIGNED Summary|Data not available for |Allow monthly harvesting |certain months should not |only so Data not available |result in a complete |for certain months do not |failure of the harvest |result in a complete | |failure of the harvest Target Milestone|--- |26.11 Assignee|koha-bugs@lists.koha-commun |matt.blenkinsop@openfifth.c |ity.org |o.uk Sponsorship status|--- |Sponsored --- Comment #2 from Michaela Sieber <michaela.sieber@kit.edu> --- Link to Omnibus Bug to track all ERM Usage Improvements Switch to monthly harvesting, so that the COUNTER log can distinguish which months had no usage. This means the user can still select a date range in the interface, but in the background this range is split into individual background jobs per month, similar to how this is already done for the report types. A migration script must be provided to retroactively bring existing entries into the new structure. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43904 [Bug 43904] [OMNIBUS] ERM Usage Improvements -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |42217 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42217 [Bug 42217] ERM Usage – Harvesting Twice Does Not Overwrite Data -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |41307 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 [Bug 41307] Improve erm_counter_logs and automatically indicate when a harvest job is completed (successfully or with an error) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |41306 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41306 [Bug 41306] ERM Usage Date picker for COUNTER data harvesting should not ask to choose a day -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Christian Nelson <christian.nelson@uwasa.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |christian.nelson@uwasa.fi -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Initiative type|--- |Epic Strategic theme|--- |User Experience -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |johanna.miettunen@haaga-hel | |ia.fi, | |jvinke@austinseminary.edu, | |nick@bywatersolutions.com, | |veli-matti.hakkinen@jyu.fi Text to go in the| |COUNTER usage harvesting release notes| |has been improved to | |process each month | |separately. This prevents | |missing or | |not-yet-available usage | |data for one month from | |causing the entire | |harvesting process to fail. | |Available data for other | |months can now be harvested | |successfully. This change | |is a prerequisite for the | |improvements introduced in | |Bug 38822. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209217|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209218|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209219|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209220|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #11 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210197 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210197&action=edit Bug 41294: Split harvest period into monthly jobs A harvest over several months ran as one background job and one SUSHI request. Providers serve usage per month, a failure in one month lost the whole range, and the counter log held one row for the range so a month could not be retried on its own. Koha::ERM::EUsage::UsageDataProvider->harvest_periods splits a date range into one period per calendar month: the first period starts on the begin date, the last ends on the end date, equal dates give one period. The dates are calendar days in the floating time zone, so the split does not depend on the server time zone or on daylight saving. An end date after today is reduced to today; a begin date after today throws BadParameter. enqueue_sushi_harvest_jobs enqueues one job per report type and month and returns the jobs with their report type and period. The process_SUSHI_response API returns that list, accepts equal dates, and answers 400 with the message of a BadParameter exception. Test plan: 1. On the data providers list, click "Run now" for a provider and ask for a period of three months. The message lists, per report type, "3 jobs queued for report type TR_J1" with one link per month 2. Click a month link: the background job page shows the report type and the harvest period of that month only, with one month of data 3. Go to E-resource management > Home: the "Latest SUSHI Counter jobs" widget shows the three jobs with a "Report type" and a "Period" column 4. Let one month fail (for example disable the network during the harvest): the failed job page names the report type and the period, and the other months still finish 5. Run the cronjob: misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 \ --end-date 2024-03-31 --dry-run It lists, per provider and report type, the three monthly periods that would be enqueued and enqueues nothing 6. Run it without --dry-run: it reports the jobs it enqueued per report type with their month range and job ids, and the jobs appear in the widget 7. Run it with --begin-date set to a date after today: the script stops with an error message and enqueues nothing 8. Run it with --end-date in the future: the end date is reduced to today and no error is reported 9. prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #12 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210198 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210198&action=edit Bug 41294: Update the job queue messages in the UI The harvest dialog reported one job. It now lists the queued jobs per report type, with one link per month to the page of that job, so a failed month can be found and retried. The latest SUSHI jobs widget on the ERM dashboard shows the report type and the period of each job; the period text is translatable. Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #13 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210199 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210199&action=edit Bug 41294: Update the job report for the background job and cronjob The background job page shows the report type and the harvest period of each monthly job, and names the period in the failure message; both strings are translatable. The cronjob reports the jobs it enqueued per report type, in the order of the provider's report types, with the month range and the job ids. --dry-run now lists the report types and the monthly periods that would be enqueued for each provider instead of printing nothing. A bad date range stops the script with the message of the exception. Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #14 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210200 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210200&action=edit Bug 41294: Update unit tests harvest_periods: one month, several months across a year boundary, a leap year, equal dates, a missing date, a begin date after the end date, an end date in the future reduced to today, an end date of today, and a begin date in the future. The clock is fixed with Time::Fake for the cases that depend on today. enqueue_sushi_harvest_jobs: one job per report type and month with the monthly dates in the job data. The API test checks the returned jobs, equal dates, the 400 answers and the permission check. The Cypress tests check the queued jobs message with one link per month, and the period column of the dashboard widget. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 --dry-run Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210197|0 |1 is obsolete| | Attachment #210198|0 |1 is obsolete| | Attachment #210199|0 |1 is obsolete| | Attachment #210200|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #15 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210237 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210237&action=edit Bug 41294: Split harvest period into monthly jobs A harvest over several months ran as one background job and one SUSHI request. Providers serve usage per month, a failure in one month lost the whole range, and the counter log held one row for the range so a month could not be retried on its own. Koha::ERM::EUsage::UsageDataProvider->harvest_periods splits a date range into one period per calendar month: the first period starts on the begin date, the last ends on the end date, equal dates give one period. The dates are calendar days in the floating time zone, so the split does not depend on the server time zone or on daylight saving. An end date after today is reduced to today; a begin date after today throws BadParameter. enqueue_sushi_harvest_jobs enqueues one job per report type and month and returns the jobs with their report type and period. The process_SUSHI_response API returns that list, accepts equal dates, and answers 400 with the message of a BadParameter exception. Test plan: 1. On the data providers list, click "Run now" for a provider and ask for a period of three months. The message lists, per report type, "3 jobs queued for report type TR_J1" with one link per month 2. Click a month link: the background job page shows the report type and the harvest period of that month only, with one month of data 3. Go to E-resource management > Home: the "Latest SUSHI Counter jobs" widget shows the three jobs with a "Report type" and a "Period" column 4. Let one month fail (for example disable the network during the harvest): the failed job page names the report type and the period, and the other months still finish 5. Run the cronjob: misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 \ --end-date 2024-03-31 --dry-run It lists, per provider and report type, the three monthly periods that would be enqueued and enqueues nothing 6. Run it without --dry-run: it reports the jobs it enqueued per report type with their month range and job ids, and the jobs appear in the widget 7. Run it with --begin-date set to a date after today: the script stops with an error message and enqueues nothing 8. Run it with --end-date in the future: the end date is reduced to today and no error is reported 9. prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #16 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210238 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210238&action=edit Bug 41294: Update the job queue messages in the UI The harvest dialog reported one job. It now lists the queued jobs per report type, with one link per month to the page of that job, so a failed month can be found and retried. The latest SUSHI jobs widget on the ERM dashboard shows the report type and the period of each job; the period text is translatable. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #17 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210239 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210239&action=edit Bug 41294: Update the job report for the background job and cronjob The background job page shows the report type and the harvest period of each monthly job, and names the period in the failure message; both strings are translatable. The cronjob reports the jobs it enqueued per report type, in the order of the provider's report types, with the month range and the job ids. --dry-run now lists the report types and the monthly periods that would be enqueued for each provider instead of printing nothing. A bad date range stops the script with the message of the exception. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #18 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210240 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210240&action=edit Bug 41294: Update unit tests harvest_periods: one month, several months across a year boundary, a leap year, equal dates, a missing date, a begin date after the end date, an end date in the future reduced to today, an end date of today, and a begin date in the future. The clock is fixed with Time::Fake for the cases that depend on today. enqueue_sushi_harvest_jobs: one job per report type and month with the monthly dates in the job data. The API test checks the returned jobs, equal dates, the 400 answers and the permission check. The Cypress tests check the queued jobs message with one link per month, and the period column of the dashboard widget. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 --dry-run Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210237|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #19 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210615 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210615&action=edit Bug 41294: Split harvest period into monthly jobs A harvest over several months ran as one background job and one SUSHI request. Providers serve usage per month, a failure in one month lost the whole range, and the counter log held one row for the range so a month could not be retried on its own. Koha::ERM::EUsage::UsageDataProvider->harvest_periods splits a date range into one period per calendar month: the first period starts on the begin date, the last ends on the end date, equal dates give one period. The dates are calendar days in the floating time zone, so the split does not depend on the server time zone or on daylight saving. An end date after today is reduced to today; a begin date after today throws BadParameter. enqueue_sushi_harvest_jobs enqueues one job per report type and month and returns the jobs with their report type and period. The process_SUSHI_response API returns that list, accepts equal dates, and answers 400 with the message of a BadParameter exception. Test plan: 1. On the data providers list, click "Run now" for a provider and ask for a period of three months. The message lists, per report type, "3 jobs queued for report type TR_J1" with one link per month 2. Click a month link: the background job page shows the report type and the harvest period of that month only, with one month of data 3. Go to E-resource management > Home: the "Latest SUSHI Counter jobs" widget shows the three jobs with a "Report type" and a "Period" column 4. Let one month fail (for example disable the network during the harvest): the failed job page names the report type and the period, and the other months still finish 5. Run the cronjob: misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 \ --end-date 2024-03-31 --dry-run It lists, per provider and report type, the three monthly periods that would be enqueued and enqueues nothing 6. Run it without --dry-run: it reports the jobs it enqueued per report type with their month range and job ids, and the jobs appear in the widget 7. Run it with --begin-date set to a date after today: the script stops with an error message and enqueues nothing 8. Run it with --end-date in the future: the end date is reduced to today and no error is reported 9. prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210240|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #20 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210616 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210616&action=edit Bug 41294: Update unit tests harvest_periods: one month, several months across a year boundary, a leap year, equal dates, a missing date, a begin date after the end date, an end date in the future reduced to today, an end date of today, and a begin date in the future. The clock is fixed with Time::Fake for the cases that depend on today. enqueue_sushi_harvest_jobs: one job per report type and month with the monthly dates in the job data. The API test checks the returned jobs, equal dates, the 400 answers and the permission check. The Cypress tests check the queued jobs message with one link per month, and the period column of the dashboard widget. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 --dry-run Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Lisa G <lgonzalez@palni.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lgonzalez@palni.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210238|0 |1 is obsolete| | Attachment #210239|0 |1 is obsolete| | Attachment #210615|0 |1 is obsolete| | Attachment #210616|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #21 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210902 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210902&action=edit Bug 41294: Split harvest period into monthly jobs A harvest over several months ran as one background job and one SUSHI request. Providers serve usage per month, a failure in one month lost the whole range, and the counter log held one row for the range so a month could not be retried on its own. Koha::ERM::EUsage::UsageDataProvider->harvest_periods splits a date range into one period per calendar month: the first period starts on the begin date, the last ends on the end date, equal dates give one period. The dates are calendar days in the floating time zone, so the split does not depend on the server time zone or on daylight saving. An end date after today is reduced to today; a begin date after today throws BadParameter. enqueue_sushi_harvest_jobs enqueues one job per report type and month and returns the jobs with their report type and period. The process_SUSHI_response API returns that list, accepts equal dates, and answers 400 with the message of a BadParameter exception. Test plan: 1. On the data providers list, click "Run now" for a provider and ask for a period of three months. The message lists, per report type, "3 jobs queued for report type TR_J1" with one link per month 2. Click a month link: the background job page shows the report type and the harvest period of that month only, with one month of data 3. Go to E-resource management > Home: the "Latest SUSHI Counter jobs" widget shows the three jobs with a "Report type" and a "Period" column 4. Let one month fail (for example disable the network during the harvest): the failed job page names the report type and the period, and the other months still finish 5. Run the cronjob: misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 \ --end-date 2024-03-31 --dry-run It lists, per provider and report type, the three monthly periods that would be enqueued and enqueues nothing 6. Run it without --dry-run: it reports the jobs it enqueued per report type with their month range and job ids, and the jobs appear in the widget 7. Run it with --begin-date set to a date after today: the script stops with an error message and enqueues nothing 8. Run it with --end-date in the future: the end date is reduced to today and no error is reported 9. prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #22 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210903 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210903&action=edit Bug 41294: Update the job queue messages in the UI The harvest dialog reported one job. It now lists the queued jobs per report type, with one link per month to the page of that job, so a failed month can be found and retried. The latest SUSHI jobs widget on the ERM dashboard shows the report type and the period of each job; the period text is translatable. Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #23 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210904 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210904&action=edit Bug 41294: Update the job report for the background job and cronjob The background job page shows the report type and the harvest period of each monthly job, and names the period in the failure message; both strings are translatable. The cronjob reports the jobs it enqueued per report type, in the order of the provider's report types, with the month range and the job ids. --dry-run now lists the report types and the monthly periods that would be enqueued for each provider instead of printing nothing. A bad date range stops the script with the message of the exception. Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #24 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210905 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210905&action=edit Bug 41294: Update unit tests harvest_periods: one month, several months across a year boundary, a leap year, equal dates, a missing date, a begin date after the end date, an end date in the future reduced to today, an end date of today, and a begin date in the future. The clock is fixed with Time::Fake for the cases that depend on today. enqueue_sushi_harvest_jobs: one job per report type and month with the monthly dates in the job data. The API test checks the returned jobs, equal dates, the 400 answers and the permission check. The Cypress tests check the queued jobs message with one link per month, and the period column of the dashboard widget. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 --dry-run Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|42217 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42217 [Bug 42217] ERM Usage – Harvesting Twice Does Not Overwrite Data -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210902|0 |1 is obsolete| | Attachment #210903|0 |1 is obsolete| | Attachment #210904|0 |1 is obsolete| | Attachment #210905|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #25 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210941 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210941&action=edit Bug 41294: Split harvest period into monthly jobs A harvest over several months ran as one background job and one SUSHI request. Providers serve usage per month, a failure in one month lost the whole range, and the counter log held one row for the range so a month could not be retried on its own. Koha::ERM::EUsage::UsageDataProvider->harvest_periods splits a date range into one period per calendar month: the first period starts on the begin date, the last ends on the end date, equal dates give one period. The dates are calendar days in the floating time zone, so the split does not depend on the server time zone or on daylight saving. An end date after today is reduced to today; a begin date after today throws BadParameter. enqueue_sushi_harvest_jobs enqueues one job per report type and month and returns the jobs with their report type and period. The process_SUSHI_response API returns that list, accepts equal dates, and answers 400 with the message of a BadParameter exception. Test plan: 1. On the data providers list, click "Run now" for a provider and ask for a period of three months. The message lists, per report type, "3 jobs queued for report type TR_J1" with one link per month 2. Click a month link: the background job page shows the report type and the harvest period of that month only, with one month of data 3. Go to E-resource management > Home: the "Latest SUSHI Counter jobs" widget shows the three jobs with a "Report type" and a "Period" column 4. Let one month fail (for example disable the network during the harvest): the failed job page names the report type and the period, and the other months still finish 5. Run the cronjob: misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 \ --end-date 2024-03-31 --dry-run It lists, per provider and report type, the three monthly periods that would be enqueued and enqueues nothing 6. Run it without --dry-run: it reports the jobs it enqueued per report type with their month range and job ids, and the jobs appear in the widget 7. Run it with --begin-date set to a date after today: the script stops with an error message and enqueues nothing 8. Run it with --end-date in the future: the end date is reduced to today and no error is reported 9. prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #26 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210942 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210942&action=edit Bug 41294: Update the job queue messages in the UI The harvest dialog reported one job. It now lists the queued jobs per report type, with one link per month to the page of that job, so a failed month can be found and retried. The latest SUSHI jobs widget on the ERM dashboard shows the report type and the period of each job; the period text is translatable. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #27 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210943 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210943&action=edit Bug 41294: Update the job report for the background job and cronjob The background job page shows the report type and the harvest period of each monthly job, and names the period in the failure message; both strings are translatable. The cronjob reports the jobs it enqueued per report type, in the order of the provider's report types, with the month range and the job ids. --dry-run now lists the report types and the monthly periods that would be enqueued for each provider instead of printing nothing. A bad date range stops the script with the message of the exception. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #28 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210944 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210944&action=edit Bug 41294: Update unit tests harvest_periods: one month, several months across a year boundary, a leap year, equal dates, a missing date, a begin date after the end date, an end date in the future reduced to today, an end date of today, and a begin date in the future. The clock is fixed with Time::Fake for the cases that depend on today. enqueue_sushi_harvest_jobs: one job per report type and month with the monthly dates in the job data. The API test checks the returned jobs, equal dates, the 400 answers and the permission check. The Cypress tests check the queued jobs message with one link per month, and the period column of the dashboard widget. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 --dry-run Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #29 from Michaela Sieber <michaela.sieber@kit.edu> --- Signed off using a sandbox -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |nick@bywatersolutions.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #30 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- I made a minor rebase here on top of 42218 and will add that as a dependency - without that bug the monthly and yearly totals can get out of alignment as a month with matching data can skip a yearly usage -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210941|0 |1 is obsolete| | Attachment #210942|0 |1 is obsolete| | Attachment #210943|0 |1 is obsolete| | Attachment #210944|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #31 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 211132 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211132&action=edit Bug 41294: Split harvest period into monthly jobs A harvest over several months ran as one background job and one SUSHI request. Providers serve usage per month, a failure in one month lost the whole range, and the counter log held one row for the range so a month could not be retried on its own. Koha::ERM::EUsage::UsageDataProvider->harvest_periods splits a date range into one period per calendar month: the first period starts on the begin date, the last ends on the end date, equal dates give one period. The dates are calendar days in the floating time zone, so the split does not depend on the server time zone or on daylight saving. An end date after today is reduced to today; a begin date after today throws BadParameter. enqueue_sushi_harvest_jobs enqueues one job per report type and month and returns the jobs with their report type and period. The process_SUSHI_response API returns that list, accepts equal dates, and answers 400 with the message of a BadParameter exception. Test plan: 1. On the data providers list, click "Run now" for a provider and ask for a period of three months. The message lists, per report type, "3 jobs queued for report type TR_J1" with one link per month 2. Click a month link: the background job page shows the report type and the harvest period of that month only, with one month of data 3. Go to E-resource management > Home: the "Latest SUSHI Counter jobs" widget shows the three jobs with a "Report type" and a "Period" column 4. Let one month fail (for example disable the network during the harvest): the failed job page names the report type and the period, and the other months still finish 5. Run the cronjob: misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 \ --end-date 2024-03-31 --dry-run It lists, per provider and report type, the three monthly periods that would be enqueued and enqueues nothing 6. Run it without --dry-run: it reports the jobs it enqueued per report type with their month range and job ids, and the jobs appear in the widget 7. Run it with --begin-date set to a date after today: the script stops with an error message and enqueues nothing 8. Run it with --end-date in the future: the end date is reduced to today and no error is reported 9. prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #32 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 211133 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211133&action=edit Bug 41294: Update the job queue messages in the UI The harvest dialog reported one job. It now lists the queued jobs per report type, with one link per month to the page of that job, so a failed month can be found and retried. The latest SUSHI jobs widget on the ERM dashboard shows the report type and the period of each job; the period text is translatable. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #33 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 211134 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211134&action=edit Bug 41294: Update the job report for the background job and cronjob The background job page shows the report type and the harvest period of each monthly job, and names the period in the failure message; both strings are translatable. The cronjob reports the jobs it enqueued per report type, in the order of the provider's report types, with the month range and the job ids. --dry-run now lists the report types and the monthly periods that would be enqueued for each provider instead of printing nothing. A bad date range stops the script with the message of the exception. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #34 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 211135 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211135&action=edit Bug 41294: Update unit tests harvest_periods: one month, several months across a year boundary, a leap year, equal dates, a missing date, a begin date after the end date, an end date in the future reduced to today, an end date of today, and a begin date in the future. The clock is fixed with Time::Fake for the cases that depend on today. enqueue_sushi_harvest_jobs: one job per report type and month with the monthly dates in the job data. The API test checks the returned jobs, equal dates, the 400 answers and the permission check. The Cypress tests check the queued jobs message with one link per month, and the period column of the dashboard widget. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/api/v1/erm_usage_data_providers.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts misc/cronjobs/erm_run_harvester.pl --begin-date 2024-01-01 --dry-run Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |42218 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 [Bug 42218] ERM Usage – Database Table erm_usage_yus -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #35 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 211612 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211612&action=edit Bug 41294: (RM follow-up) Fix Dialog_spec expected message Test plan: 1) yarn cypress run --spec t/cypress/integration/ERM/Dialog_spec.ts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |26.11.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 --- Comment #36 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Thanks everyone! Pushed to main for 26.11! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org