[Bug 41307] New: Automatically indicate when a harvest job is completed (successfully or with an error)
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Bug ID: 41307 Summary: Automatically indicate when a harvest job is completed (successfully or with an error) Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: ERM Assignee: jonathan.druart@gmail.com Reporter: mathsabypro@gmail.com QA Contact: testopia@bugs.koha-community.org CC: jonathan.druart@gmail.com, jonathan.field@openfifth.co.uk, martin.renvoize@openfifth.co.uk, matt.blenkinsop@openfifth.co.uk, pedro.amorim@openfifth.co.uk In /erm/eusage/usage_data_providers when you run an import for a data source, the list of corresponding jobs is displayed above the table ("Job for report type DR has been queued. Check job progress.") It would be nice if this information were automatically updated once the job is completed ("Job for report type DR is completed. Check job details.") This may not be technically possible ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #1 from Mathieu Saby <mathsabypro@gmail.com> --- I see this information will be added in the widget on the ERM home page https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39320 But it could be added too in erm/eusage/usage_data_providers page -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew.auld@openfifth.co.uk | |, clemens.tubach@kit.edu, | |michaela.sieber@kit.edu --- Comment #2 from Michaela Sieber <michaela.sieber@kit.edu> --- As far as I can see the widget only displays if the job is still running or completed / cancelled. What you would like to see is the error code and message. This is the SQL-Statement which gives you the option to find the background jobs type erm_sushi_harvester that ended with an error (for example if no data were available or if the request was queued, etc.) : SELECT CONCAT('<a target="_blank" href=\"/cgi-bin/koha/admin/background_jobs.pl?op=view&id=',bj.id,'\">', bj.id, '</a>' ) AS "Job-ID", bj.enqueued_on AS 'Date/time', CONCAT('<a target="_blank" href=\"/cgi-bin/koha/erm/eusage/usage_data_providers/', JSON_VALUE(bj.data,'$.ud_provider_id'), '\">', JSON_VALUE(bj.data,'$.ud_provider_id'), '</a>' ) AS "Provider-ID", JSON_VALUE(bj.data,'$.ud_provider_name') AS 'provider name', JSON_VALUE(bj.data,'$.report_type') AS 'Report', JSON_VALUE(bj.data,'$.begin_date') AS 'report begin_date', JSON_VALUE(bj.data,'$.end_date') AS 'report end_date', JSON_VALUE(bj.data,'$.messages[0].code') AS 'error code', JSON_VALUE(bj.data,'$.messages[0].message') AS 'error message' FROM background_jobs bj WHERE type='erm_sushi_harvester' AND JSON_VALUE(bj.data,'$.messages[0].type')='error' /* #All jobs with type error in data column, e.g. No Usage Available for Requested Dates (3030) or Usage Not Ready for Requested Dates (3031) or Fatal - Service Not Available (1000) or Error - Requestor Not Authorized to Access Service (2000) */ AND DATE(bj.enqueued_on) BETWEEN <<from|date>> AND <<to|date>> ORDER BY bj.enqueued_on DESC -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41223 --- Comment #3 from Michaela Sieber <michaela.sieber@kit.edu> --- We need more information about the harvesting jobs and store them for example in the existing database table for erm_counter_logs. Relevant information are error codes, harvested month, background-job-ID. This would also be the groundwork for a Retry management of harvesting jobs that were finished with error codes like 1011 = "Report Queued for Processing" or 1020 = "Client has made too many requests" See also Bug 41223 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|jonathan.druart@gmail.com |matt.blenkinsop@openfifth.c | |o.uk Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |26.11 Blocks| |43904 --- Comment #4 from Michaela Sieber <michaela.sieber@kit.edu> --- Link to Omnibus Bug to track all ERM Usage Improvements Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43904 [Bug 43904] [OMNIBUS] ERM Usage Improvements -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Automatically indicate when |Improve erm_counter_logs |a harvest job is completed |and automatically indicate |(successfully or with an |when a harvest job is |error) |completed (successfully or | |with an error) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |41294 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 [Bug 41294] Allow monthly harvesting only so Data not available for certain months do not result in a complete failure of the harvest -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |38822 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 [Bug 38822] ERM Usage data provider summary should be more complete (all reports and all months) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209179|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #12 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 209225 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209225&action=edit Bug 41307: Record the outcome of every SUSHI harvest in the counter logs Until now a counter log row was only written after a COUNTER file was stored, and logdetails was always empty. A harvest that failed left no record, and any exception in the SUSHI response blocked the import, including warnings such as 3040 Partial Data Returned. This patch writes one counter log row for every harvest attempt and for every file upload: - harvest_sushi collects the exceptions of the SUSHI response (top level exception, Exceptions array, Report_Header.Exceptions, vendor error bodies) and classifies the outcome as success, no_usage, no_data, retry or error. Code 3030 No Usage Available without items gives no_usage: the month is complete, the usage is zero. Code 3032, HTTP 204 and an empty report give no_data. Retry codes are 1000, 1010, 1011, 1020, 3031 and 3040, plus HTTP 202, 429 and 5xx. The request parameter warnings 3050, 3060, 3061 and 3062 are informational: the report is complete. A response that is not JSON is an error. - Reports with items are stored even when they carry a warning exception; the log is flagged retry for 3040 and 3031. A report with items and an error exception is not imported: it is logged as error. - logdetails holds a JSON document with the HTTP status, the exceptions (code, severity, message, data), an optional free text message and the COUNTER release. The job messages carry the exception message and data. The request URL is never stored or logged, it holds the API key. The HTTP failure is logged through Koha::Logger instead of warn. - A harvest without a file has a log row without a file name. One row per background job: a failure after the row was written does not add one. - CounterFile::store accepts the log fields of a harvest. For an uploaded file it reads the report type, the period and the Exceptions row from the file header. - CounterLog gains sushi_code_status, decoded_logdetails and a background_job accessor. - The background job passes its id to the harvest and reports the harvest status, shown on the job detail page with a badge per status. Test plan: 1. Run updatedatabase. Open a provider that had import logs before the patch, tab "Import logs": the old rows show status "Success" 2. Run a harvest over two months. On the Import logs tab, check one row per report type and month with the columns Filename, Report type, Period, Status, Details, Import date, Imported by and Job. The job link opens the background job page, which shows a "Harvest status" badge 3. Create a provider with wrong credentials and run a harvest: the job finishes, the Import logs tab shows a row with status "Error", no file name, the exception code and message in Details, and no Download or Delete action. Change the URL to a host that does not exist and run a harvest again: the new row shows status "Retry" with HTTP 500 and the connection error in Details 4. Harvest a month for which the provider has no usage (if the provider allows it): the row shows status "No usage" or "No data" and the job page shows the same badge 5. Use the Status filter of the table: select "Error", then "Success" and check the rows shown 6. Upload a COUNTER file on the Manual upload tab: its row shows the report type and the period read from the file header, status "Success", Download and Delete actions, and no job link 7. Delete the uploaded file: the row disappears 8. Age the harvest jobs so the cleanup can purge them, in koha-mysql: UPDATE background_jobs SET ended_on = DATE_SUB(NOW(), INTERVAL 2 DAY) WHERE type = 'erm_sushi_harvester'; Then run misc/cronjobs/cleanup_database.pl --jobs-days 1 --jobs-type erm_sushi_harvester --confirm. The import log rows stay, with an empty Job column, and the job link is gone 9. prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #13 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 209467 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209467&action=edit Bug 41307: Backfill the period of existing counter logs and remove duplicated usage The counter log rows written before this patch have no report type and no period, so the data providers summary skips every month harvested before the upgrade, and the Import logs tab shows those rows without a report type or a period. Re-harvests before Bug 42217 also left duplicated rows in erm_usage_mus, which double the yearly totals. The atomicupdate now backfills the rows that have no period: - the report type is the one of the counter file - the months come from the Reporting_Period header row of the file, or from its month columns when the file has no such row. Before Bug 41294 one harvest covered several months, so such a row becomes one row per month, all with the same file, import date, user and status - every month gets status success: the old code only wrote a log row after a file was imported - the background job is not linked A row whose file has no reporting period is reported and left without a period. The backfill only reads the first 4000 bytes of each file and runs again without changes. It then removes the duplicated monthly usage rows, with the identity of Bug 42217 (every column except usage_count). The newest row is kept: the harvest path of 42217 keeps the lowest id because it overwrites the count, but here there is no harvested count, and the newest row holds the last count the provider sent. Test plan: 1. On a database with harvests made before this patch, or after UPDATE erm_counter_logs SET report_type = NULL, report_year = NULL, report_month = NULL; on a few rows, run updatedatabase: it reports the rows updated and added 2. Open the provider, Import logs tab: every old row now shows its report type and period; a file that covered several months has one row per month, all with the same file name and Download link 3. Open the Summary page: the old months show "Success" 4. Run updatedatabase again: no counter log is changed 5. Duplicate a row of erm_usage_mus with a different usage_count and run updatedatabase: the older row is removed and the yearly total of the title is correct again Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |COUNTER error handling and release notes| |logging in the ERM module | |have been significantly | |improved. Previously, | |unsuccessful harvest | |attempts were not logged, | |and the log details did not | |contain useful information | |about COUNTER errors. Now, | |every harvest attempt and | |file upload generates a log | |entry, including attempts | |with COUNTER errors. | | | |COUNTER exceptions returned | |by the COUNTER API are now | |captured and logged in | |detail, including their | |error codes, together with | |the HTTP status and COUNTER | |release. This makes it much | |easier to identify, | |understand and troubleshoot | |harvesting problems. | |Relevant error information | |is also included in the | |background job messages and | |displayed on the job detail | |page. | | | |The handling of COUNTER | |exceptions has also been | |improved. Warnings such as | |3040 Partial Data Returned | |no longer automatically | |prevent reports containing | |usage data from being | |imported. Informational | |warnings are distinguished | |from errors, and retryable | |(e.g. Error 1011) | |conditions are identified | |separately. Responses | |indicating that no usage | |(Error 3030) is available | |are also distinguished from | |missing data and actual | |errors. | | | |Background jobs now report | |the actual harvest outcome, | |with status badges | |indicating success, no | |usage, no data, retry or | |error. | | | |Together with the monthly | |harvesting improvements | |introduced in bug 41294 and | |the comprehensive monthly | |usage overview addressed in | |bug 38822, these changes | |provide more reliable | |COUNTER harvesting and | |significantly better | |insight into harvesting | |errors and their causes. CC| |christian.nelson@uwasa.fi, | |johanna.miettunen@haaga-hel | |ia.fi, | |nick@bywatersolutions.com, | |veli-matti.hakkinen@jyu.fi -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209176|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209177|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209178|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209180|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209181|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209182|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209225|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #14 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210201 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210201&action=edit Bug 41307: Add status, background_job_id and report period to erm_counter_logs A counter log row was only written after a COUNTER file was stored, so a failed harvest left no trace and a log could not be tied to the job that produced it. This adds to erm_counter_logs: - background_job_id, a foreign key to background_jobs with ON DELETE SET NULL, so a log survives the purge of its job by cleanup_database - report_type, report_year and report_month, the harvested period of a single-month job, so the import logs and the summary can be read per month - status, the outcome of the harvest or import: success, no_usage, no_data, retry or error, with an index for the status filter Existing rows get status success: before this patch a row was only written after a successful import. The counter log API definition gains the same fields. Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #15 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210202 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210202&action=edit Bug 41307: DBIC schema changes Generated for the new erm_counter_logs columns. The relationship between the counter log and its background job is added above the marker by the loader (belongs_to background_job, LEFT JOIN, on_delete SET NULL) and BackgroundJob gains has_many erm_counter_logs. Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #16 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210203 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210203&action=edit Bug 41307: Tolerate COUNTER header exceptions without Data The CSV header builders for COUNTER 5 and 5.1 concatenated Code, Message and Data of every Report_Header exception. An exception without Data (or without Message) raised an "uninitialized value" warning. Reports that carry a warning exception such as 3040 Partial Data Returned are now stored, so the header must handle these fields when they are absent. Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #17 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210204 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210204&action=edit Bug 41307: Record the outcome of every SUSHI harvest in the counter logs Until now a counter log row was only written after a COUNTER file was stored, and logdetails was always empty. A harvest that failed left no record, and any exception in the SUSHI response blocked the import, including warnings such as 3040 Partial Data Returned. This patch writes one counter log row for every harvest attempt and for every file upload: - harvest_sushi collects the exceptions of the SUSHI response (top level exception, Exceptions array, Report_Header.Exceptions, vendor error bodies) and classifies the outcome as success, no_usage, no_data, retry or error. Code 3030 No Usage Available without items gives no_usage: the month is complete, the usage is zero. Code 3032, HTTP 204 and an empty report give no_data. Retry codes are 1000, 1010, 1011, 1020, 3031 and 3040, plus HTTP 202, 429 and 5xx. The request parameter warnings 3050, 3060, 3061 and 3062 are informational: the report is complete. A response that is not JSON is an error. - Reports with items are stored even when they carry a warning exception; the log is flagged retry for 3040 and 3031. A report with items and an error exception is not imported: it is logged as error. - logdetails holds a JSON document with the HTTP status, the exceptions (code, severity, message, data), an optional free text message and the COUNTER release. The job messages carry the exception message and data. The request URL is never stored or logged, it holds the API key. The HTTP failure is logged through Koha::Logger instead of warn. - A harvest without a file has a log row without a file name. One row per background job: a failure after the row was written does not add one. - CounterFile::store accepts the log fields of a harvest. For an uploaded file it reads the report type, the period and the Exceptions row from the file header. - CounterLog gains sushi_code_status, decoded_logdetails and a background_job accessor. - The background job passes its id to the harvest and reports the harvest status, shown on the job detail page with a badge per status. Test plan: 1. Run updatedatabase. Open a provider that had import logs before the patch, tab "Import logs": the old rows show status "Success" 2. Run a harvest over two months. On the Import logs tab, check one row per report type and month with the columns Filename, Report type, Period, Status, Details, Import date, Imported by and Job. The job link opens the background job page, which shows a "Harvest status" badge 3. Create a provider with wrong credentials and run a harvest: the job finishes, the Import logs tab shows a row with status "Error", no file name, the exception code and message in Details, and no Download or Delete action. Change the URL to a host that does not exist and run a harvest again: the new row shows status "Retry" with HTTP 500 and the connection error in Details 4. Harvest a month for which the provider has no usage (if the provider allows it): the row shows status "No usage" or "No data" and the job page shows the same badge 5. Use the Status filter of the table: select "Error", then "Success" and check the rows shown 6. Upload a COUNTER file on the Manual upload tab: its row shows the report type and the period read from the file header, status "Success", Download and Delete actions, and no job link 7. Delete the uploaded file: the row disappears 8. Age the harvest jobs so the cleanup can purge them, in koha-mysql: UPDATE background_jobs SET ended_on = DATE_SUB(NOW(), INTERVAL 2 DAY) WHERE type = 'erm_sushi_harvester'; Then run misc/cronjobs/cleanup_database.pl --jobs-days 1 --jobs-type erm_sushi_harvester --confirm. The import log rows stay, with an empty Job column, and the job link is gone 9. prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #18 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210205 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210205&action=edit Bug 41307: Expose status and the background job on the counter logs API The counter log definition gains status, and the list endpoint can embed background_job. Filtering on status already works through the q parameter, for example q={"status":"retry"}. Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #19 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210206 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210206&action=edit Bug 41307: Show harvest status in the import logs tab The Import logs tab of a usage data provider now shows the report type, the period, a status badge with a select filter, the exception details and a link to the background job. Download and delete are only offered for rows that have a counter file. The empty state counts counter logs instead of counter files, so failed harvests are listed too. Test plan: 1. Run a harvest for a provider over two months 2. Open the provider, Import logs tab 3. Check one row per report type and month, with status, period and a job link 4. Filter the status column on "Retry needed" 5. Upload a COUNTER file and check its row Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #20 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210207 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210207&action=edit Bug 41307: Unit tests CounterLog.t covers sushi_code_status, decoded_logdetails and the background_job accessor. UsageDataProvider.t traces every outcome of harvest_sushi: items with and without exceptions, the retry, no_usage, no_data and error codes, a request parameter warning next to items, an error exception next to items (not imported), HTTP failures, a response that is not JSON, a file too large and an unsupported release; the log row, its details and the job messages are checked for each. The harvester job tests check the status in the job report and the log row of the job. The older harvester test keeps its message format: message and data, without the severity. The API test covers the status filter and the background_job embed. The Cypress test covers the import logs tab. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #21 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210208 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210208&action=edit Bug 41307: Tell administrators to run the counter data migration script The atomicupdate adds the status, report type and report period columns to erm_counter_logs. It does not change any existing row. The counter log rows written before this patch have no report type and no period, so the data providers summary skips every month harvested before the upgrade. Re-harvests before Bug 42217 also left duplicated rows in erm_usage_mus, which double the yearly totals. The script misc/maintenance/erm_migrate_counter_data.pl, added by the next patch, fixes both. The atomicupdate ends with a message that names it, so the upgrade does not have to read every stored counter file. Test plan: 1. On a database from before this patch, run updatedatabase: the columns, the index and the foreign key are added, and a message tells you to run misc/maintenance/erm_migrate_counter_data.pl 2. Check that no row of erm_counter_logs or erm_usage_mus has changed Assisted-by: Sonnet 5.5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #22 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210209 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210209&action=edit Bug 41307: Add a maintenance script to migrate the old counter data The counter log rows written before Bug 41307 have no report type and no period, so the data providers summary skips every month harvested before the upgrade, and the Import logs tab shows those rows without a report type or a period. Re-harvests before Bug 42217 also left duplicated rows in erm_usage_mus, which double the yearly totals. misc/maintenance/erm_migrate_counter_data.pl fixes both. Without --confirm it is a dry run: it reports what it would change and writes nothing. --logs and --usage run one part only, --verbose reports each row. The script backfills the rows that have no period: - the report type is the one of the counter file - the months come from the Reporting_Period header row of the file, or from its month columns when the file has no such row. Before Bug 41294 one harvest covered several months, so such a row becomes one row per month, all with the same file, import date, user and status - every month gets status success: the old code only wrote a log row after a file was imported - the background job is not linked A row whose file has no reporting period is reported and left without a period. The script only reads the first 4000 bytes of each file and can run again without changes. It then removes the duplicated monthly usage rows, with the identity of Bug 42217 (every column except usage_count). The newest row is kept: the harvest path of 42217 keeps the lowest id because it overwrites the count, but here there is no harvested count, and the newest row holds the last count the provider sent. Each part runs in one transaction. Test plan: 1. On a database with harvests made before this patch, or after UPDATE erm_counter_logs SET report_type = NULL, report_year = NULL, report_month = NULL; on a few rows, run perl misc/maintenance/erm_migrate_counter_data.pl -v It lists the rows it would update and add, and changes nothing 2. Run it again with --confirm: it reports the rows updated and added 3. Open the provider, Import logs tab: every old row now shows its report type and period; a file that covered several months has one row per month, all with the same file name and Download link 4. Open the Summary page: the old months show "Success" 5. Run the script with --confirm again: no counter log is changed 6. Duplicate a row of erm_usage_mus with a different usage_count and run the script with --usage --confirm: the older row is removed and the yearly total of the title is correct again Assisted-by: Sonnet 5.5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209467|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210201|0 |1 is obsolete| | Attachment #210202|0 |1 is obsolete| | Attachment #210203|0 |1 is obsolete| | Attachment #210204|0 |1 is obsolete| | Attachment #210205|0 |1 is obsolete| | Attachment #210206|0 |1 is obsolete| | Attachment #210207|0 |1 is obsolete| | Attachment #210208|0 |1 is obsolete| | Attachment #210209|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #23 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210257 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210257&action=edit Bug 41307: Add status, background_job_id and report period to erm_counter_logs A counter log row was only written after a COUNTER file was stored, so a failed harvest left no trace and a log could not be tied to the job that produced it. This adds to erm_counter_logs: - background_job_id, a foreign key to background_jobs with ON DELETE SET NULL, so a log survives the purge of its job by cleanup_database - report_type, report_year and report_month, the harvested period of a single-month job, so the import logs and the summary can be read per month - status, the outcome of the harvest or import: success, no_usage, no_data, retry or error, with an index for the status filter Existing rows get status success: before this patch a row was only written after a successful import. The counter log API definition gains the same fields. Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #24 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210258 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210258&action=edit Bug 41307: DBIC schema changes Generated for the new erm_counter_logs columns. The relationship between the counter log and its background job is added above the marker by the loader (belongs_to background_job, LEFT JOIN, on_delete SET NULL) and BackgroundJob gains has_many erm_counter_logs. Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #25 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210259 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210259&action=edit Bug 41307: Tolerate COUNTER header exceptions without Data The CSV header builders for COUNTER 5 and 5.1 concatenated Code, Message and Data of every Report_Header exception. An exception without Data (or without Message) raised an "uninitialized value" warning. Reports that carry a warning exception such as 3040 Partial Data Returned are now stored, so the header must handle these fields when they are absent. Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #26 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210260 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210260&action=edit Bug 41307: Record the outcome of every SUSHI harvest in the counter logs Until now a counter log row was only written after a COUNTER file was stored, and logdetails was always empty. A harvest that failed left no record, and any exception in the SUSHI response blocked the import, including warnings such as 3040 Partial Data Returned. This patch writes one counter log row for every harvest attempt and for every file upload: - harvest_sushi collects the exceptions of the SUSHI response (top level exception, Exceptions array, Report_Header.Exceptions, vendor error bodies) and classifies the outcome as success, no_usage, no_data, retry or error. Code 3030 No Usage Available without items gives no_usage: the month is complete, the usage is zero. Code 3032, HTTP 204 and an empty report give no_data. Retry codes are 1000, 1010, 1011, 1020, 3031 and 3040, plus HTTP 202, 429 and 5xx. The request parameter warnings 3050, 3060, 3061 and 3062 are informational: the report is complete. A response that is not JSON is an error. - Reports with items are stored even when they carry a warning exception; the log is flagged retry for 3040 and 3031. A report with items and an error exception is not imported: it is logged as error. - logdetails holds a JSON document with the HTTP status, the exceptions (code, severity, message, data), an optional free text message and the COUNTER release. The job messages carry the exception message and data. The request URL is never stored or logged, it holds the API key. The HTTP failure is logged through Koha::Logger instead of warn. - A harvest without a file has a log row without a file name. One row per background job: a failure after the row was written does not add one. - CounterFile::store accepts the log fields of a harvest. For an uploaded file it reads the report type, the period and the Exceptions row from the file header. - CounterLog gains sushi_code_status, decoded_logdetails and a background_job accessor. - The background job passes its id to the harvest and reports the harvest status, shown on the job detail page with a badge per status. Test plan: 1. Run updatedatabase. Open a provider that had import logs before the patch, tab "Import logs": the old rows show status "Success" 2. Run a harvest over two months. On the Import logs tab, check one row per report type and month with the columns Filename, Report type, Period, Status, Details, Import date, Imported by and Job. The job link opens the background job page, which shows a "Harvest status" badge 3. Create a provider with wrong credentials and run a harvest: the job finishes, the Import logs tab shows a row with status "Error", no file name, the exception code and message in Details, and no Download or Delete action. Change the URL to a host that does not exist and run a harvest again: the new row shows status "Retry" with HTTP 500 and the connection error in Details 4. Harvest a month for which the provider has no usage (if the provider allows it): the row shows status "No usage" or "No data" and the job page shows the same badge 5. Use the Status filter of the table: select "Error", then "Success" and check the rows shown 6. Upload a COUNTER file on the Manual upload tab: its row shows the report type and the period read from the file header, status "Success", Download and Delete actions, and no job link 7. Delete the uploaded file: the row disappears 8. Age the harvest jobs so the cleanup can purge them, in koha-mysql: UPDATE background_jobs SET ended_on = DATE_SUB(NOW(), INTERVAL 2 DAY) WHERE type = 'erm_sushi_harvester'; Then run misc/cronjobs/cleanup_database.pl --jobs-days 1 --jobs-type erm_sushi_harvester --confirm. The import log rows stay, with an empty Job column, and the job link is gone 9. prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #27 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210261 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210261&action=edit Bug 41307: Expose status and the background job on the counter logs API The counter log definition gains status, and the list endpoint can embed background_job. Filtering on status already works through the q parameter, for example q={"status":"retry"}. Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #28 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210262 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210262&action=edit Bug 41307: Show harvest status in the import logs tab The Import logs tab of a usage data provider now shows the report type, the period, a status badge with a select filter, the exception details and a link to the background job. Download and delete are only offered for rows that have a counter file. The empty state counts counter logs instead of counter files, so failed harvests are listed too. Test plan: 1. Run a harvest for a provider over two months 2. Open the provider, Import logs tab 3. Check one row per report type and month, with status, period and a job link 4. Filter the status column on "Retry needed" 5. Upload a COUNTER file and check its row Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #29 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210263 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210263&action=edit Bug 41307: Unit tests CounterLog.t covers sushi_code_status, decoded_logdetails and the background_job accessor. UsageDataProvider.t traces every outcome of harvest_sushi: items with and without exceptions, the retry, no_usage, no_data and error codes, a request parameter warning next to items, an error exception next to items (not imported), HTTP failures, a response that is not JSON, a file too large and an unsupported release; the log row, its details and the job messages are checked for each. The harvester job tests check the status in the job report and the log row of the job. The older harvester test keeps its message format: message and data, without the severity. The API test covers the status filter and the background_job embed. The Cypress test covers the import logs tab. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #30 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210264 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210264&action=edit Bug 41307: Tell administrators to run the counter data migration script The atomicupdate adds the status, report type and report period columns to erm_counter_logs. It does not change any existing row. The counter log rows written before this patch have no report type and no period, so the data providers summary skips every month harvested before the upgrade. Re-harvests before Bug 42217 also left duplicated rows in erm_usage_mus, which double the yearly totals. The script misc/maintenance/erm_migrate_counter_data.pl, added by the next patch, fixes both. The atomicupdate ends with a message that names it, so the upgrade does not have to read every stored counter file. Test plan: 1. On a database from before this patch, run updatedatabase: the columns, the index and the foreign key are added, and a message tells you to run misc/maintenance/erm_migrate_counter_data.pl 2. Check that no row of erm_counter_logs or erm_usage_mus has changed Assisted-by: Sonnet 5.5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #31 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210265 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210265&action=edit Bug 41307: Add a maintenance script to migrate the old counter data The counter log rows written before Bug 41307 have no report type and no period, so the data providers summary skips every month harvested before the upgrade, and the Import logs tab shows those rows without a report type or a period. Re-harvests before Bug 42217 also left duplicated rows in erm_usage_mus, which double the yearly totals. misc/maintenance/erm_migrate_counter_data.pl fixes both. Without --confirm it is a dry run: it reports what it would change and writes nothing. --logs and --usage run one part only, --verbose reports each row. The script backfills the rows that have no period: - the report type is the one of the counter file - the months come from the Reporting_Period header row of the file, or from its month columns when the file has no such row. Before Bug 41294 one harvest covered several months, so such a row becomes one row per month, all with the same file, import date, user and status - every month gets status success: the old code only wrote a log row after a file was imported - the background job is not linked A row whose file has no reporting period is reported and left without a period. The script only reads the first 4000 bytes of each file and can run again without changes. It then removes the duplicated monthly usage rows, with the identity of Bug 42217 (every column except usage_count). The newest row is kept: the harvest path of 42217 keeps the lowest id because it overwrites the count, but here there is no harvested count, and the newest row holds the last count the provider sent. Each part runs in one transaction. Test plan: 1. On a database with harvests made before this patch, or after UPDATE erm_counter_logs SET report_type = NULL, report_year = NULL, report_month = NULL; on a few rows, run perl misc/maintenance/erm_migrate_counter_data.pl -v It lists the rows it would update and add, and changes nothing 2. Run it again with --confirm: it reports the rows updated and added 3. Open the provider, Import logs tab: every old row now shows its report type and period; a file that covered several months has one row per month, all with the same file name and Download link 4. Open the Summary page: the old months show "Success" 5. Run the script with --confirm again: no counter log is changed 6. Duplicate a row of erm_usage_mus with a different usage_count and run the script with --usage --confirm: the older row is removed and the yearly total of the title is correct again Assisted-by: Sonnet 5.5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210257|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210258|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210259|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210260|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210261|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210262|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210263|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210264|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210265|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #32 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210275 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210275&action=edit Bug 41307: Add status, background_job_id and report period to erm_counter_logs A counter log row was only written after a COUNTER file was stored, so a failed harvest left no trace and a log could not be tied to the job that produced it. This adds to erm_counter_logs: - background_job_id, a foreign key to background_jobs with ON DELETE SET NULL, so a log survives the purge of its job by cleanup_database - report_type, report_year and report_month, the harvested period of a single-month job, so the import logs and the summary can be read per month - status, the outcome of the harvest or import: success, no_usage, no_data, retry or error, with an index for the status filter Existing rows get status success: before this patch a row was only written after a successful import. The counter log API definition gains the same fields. Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #33 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210276 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210276&action=edit Bug 41307: DBIC schema changes Generated for the new erm_counter_logs columns. The relationship between the counter log and its background job is added above the marker by the loader (belongs_to background_job, LEFT JOIN, on_delete SET NULL) and BackgroundJob gains has_many erm_counter_logs. Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #34 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210277 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210277&action=edit Bug 41307: Tolerate COUNTER header exceptions without Data The CSV header builders for COUNTER 5 and 5.1 concatenated Code, Message and Data of every Report_Header exception. An exception without Data (or without Message) raised an "uninitialized value" warning. Reports that carry a warning exception such as 3040 Partial Data Returned are now stored, so the header must handle these fields when they are absent. Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #35 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210278 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210278&action=edit Bug 41307: Record the outcome of every SUSHI harvest in the counter logs Until now a counter log row was only written after a COUNTER file was stored, and logdetails was always empty. A harvest that failed left no record, and any exception in the SUSHI response blocked the import, including warnings such as 3040 Partial Data Returned. This patch writes one counter log row for every harvest attempt and for every file upload: - harvest_sushi collects the exceptions of the SUSHI response (top level exception, Exceptions array, Report_Header.Exceptions, vendor error bodies) and classifies the outcome as success, no_usage, no_data, retry or error. Code 3030 No Usage Available without items gives no_usage: the month is complete, the usage is zero. Code 3032, HTTP 204 and an empty report give no_data. Retry codes are 1000, 1010, 1011, 1020, 3031 and 3040, plus HTTP 202, 429 and 5xx. The request parameter warnings 3050, 3060, 3061 and 3062 are informational: the report is complete. A response that is not JSON is an error. - Reports with items are stored even when they carry a warning exception; the log is flagged retry for 3040 and 3031. A report with items and an error exception is not imported: it is logged as error. - logdetails holds a JSON document with the HTTP status, the exceptions (code, severity, message, data), an optional free text message and the COUNTER release. The job messages carry the exception message and data. The request URL is never stored or logged, it holds the API key. The HTTP failure is logged through Koha::Logger instead of warn. - A harvest without a file has a log row without a file name. One row per background job: a failure after the row was written does not add one. - CounterFile::store accepts the log fields of a harvest. For an uploaded file it reads the report type, the period and the Exceptions row from the file header. - CounterLog gains sushi_code_status, decoded_logdetails and a background_job accessor. - The background job passes its id to the harvest and reports the harvest status, shown on the job detail page with a badge per status. Test plan: 1. Run updatedatabase. Open a provider that had import logs before the patch, tab "Import logs": the old rows show status "Success" 2. Run a harvest over two months. On the Import logs tab, check one row per report type and month with the columns Filename, Report type, Period, Status, Details, Import date, Imported by and Job. The job link opens the background job page, which shows a "Harvest status" badge 3. Create a provider with wrong credentials and run a harvest: the job finishes, the Import logs tab shows a row with status "Error", no file name, the exception code and message in Details, and no Download or Delete action. Change the URL to a host that does not exist and run a harvest again: the new row shows status "Retry" with HTTP 500 and the connection error in Details 4. Harvest a month for which the provider has no usage (if the provider allows it): the row shows status "No usage" or "No data" and the job page shows the same badge 5. Use the Status filter of the table: select "Error", then "Success" and check the rows shown 6. Upload a COUNTER file on the Manual upload tab: its row shows the report type and the period read from the file header, status "Success", Download and Delete actions, and no job link 7. Delete the uploaded file: the row disappears 8. Age the harvest jobs so the cleanup can purge them, in koha-mysql: UPDATE background_jobs SET ended_on = DATE_SUB(NOW(), INTERVAL 2 DAY) WHERE type = 'erm_sushi_harvester'; Then run misc/cronjobs/cleanup_database.pl --jobs-days 1 --jobs-type erm_sushi_harvester --confirm. The import log rows stay, with an empty Job column, and the job link is gone 9. prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #36 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210279 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210279&action=edit Bug 41307: Expose status and the background job on the counter logs API The counter log definition gains status, and the list endpoint can embed background_job. Filtering on status already works through the q parameter, for example q={"status":"retry"}. Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #37 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210280 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210280&action=edit Bug 41307: Show harvest status in the import logs tab The Import logs tab of a usage data provider now shows the report type, the period, a status badge with a select filter, the exception details and a link to the background job. Download and delete are only offered for rows that have a counter file. The empty state counts counter logs instead of counter files, so failed harvests are listed too. Test plan: 1. Run a harvest for a provider over two months 2. Open the provider, Import logs tab 3. Check one row per report type and month, with status, period and a job link 4. Filter the status column on "Retry needed" 5. Upload a COUNTER file and check its row Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #38 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210281 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210281&action=edit Bug 41307: Unit tests CounterLog.t covers sushi_code_status, decoded_logdetails and the background_job accessor. UsageDataProvider.t traces every outcome of harvest_sushi: items with and without exceptions, the retry, no_usage, no_data and error codes, a request parameter warning next to items, an error exception next to items (not imported), HTTP failures, a response that is not JSON, a file too large and an unsupported release; the log row, its details and the job messages are checked for each. The harvester job tests check the status in the job report and the log row of the job. The older harvester test keeps its message format: message and data, without the severity. The API test covers the status filter and the background_job embed. The Cypress test covers the import logs tab. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #39 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210282 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210282&action=edit Bug 41307: Tell administrators to run the counter data migration script The atomicupdate adds the status, report type and report period columns to erm_counter_logs. It does not change any existing row. The counter log rows written before this patch have no report type and no period, so the data providers summary skips every month harvested before the upgrade. Re-harvests before Bug 42217 also left duplicated rows in erm_usage_mus, which double the yearly totals. The script misc/maintenance/erm_migrate_counter_data.pl, added by the next patch, fixes both. The atomicupdate ends with a message that names it, so the upgrade does not have to read every stored counter file. Test plan: 1. On a database from before this patch, run updatedatabase: the columns, the index and the foreign key are added, and a message tells you to run misc/maintenance/erm_migrate_counter_data.pl 2. Check that no row of erm_counter_logs or erm_usage_mus has changed Assisted-by: Sonnet 5.5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #40 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210283 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210283&action=edit Bug 41307: Add a maintenance script to migrate the old counter data The counter log rows written before Bug 41307 have no report type and no period, so the data providers summary skips every month harvested before the upgrade, and the Import logs tab shows those rows without a report type or a period. Re-harvests before Bug 42217 also left duplicated rows in erm_usage_mus, which double the yearly totals. misc/maintenance/erm_migrate_counter_data.pl fixes both. Without --confirm it is a dry run: it reports what it would change and writes nothing. --logs and --usage run one part only, --verbose reports each row. The script backfills the rows that have no period: - the report type is the one of the counter file - the months come from the Reporting_Period header row of the file, or from its month columns when the file has no such row. Before Bug 41294 one harvest covered several months, so such a row becomes one row per month, all with the same file, import date, user and status - every month gets status success: the old code only wrote a log row after a file was imported - the background job is not linked A row whose file has no reporting period is reported and left without a period. The script only reads the first 4000 bytes of each file and can run again without changes. It then removes the duplicated monthly usage rows, with the identity of Bug 42217 (every column except usage_count). The newest row is kept: the harvest path of 42217 keeps the lowest id because it overwrites the count, but here there is no harvested count, and the newest row holds the last count the provider sent. The yearly usage in erm_usage_yus is then rebuilt from the monthly usage, so its totals follow the cleaned rows. This runs even when no duplicate is found. Each part runs in one transaction. Test plan: 1. On a database with harvests made before this patch, or after UPDATE erm_counter_logs SET report_type = NULL, report_year = NULL, report_month = NULL; on a few rows, run perl misc/maintenance/erm_migrate_counter_data.pl -v It lists the rows it would update and add, and changes nothing 2. Run it again with --confirm: it reports the rows updated and added 3. Open the provider, Import logs tab: every old row now shows its report type and period; a file that covered several months has one row per month, all with the same file name and Download link 4. Open the Summary page: the old months show "Success" 5. Run the script with --confirm again: no counter log is changed 6. Duplicate a row of erm_usage_mus with a different usage_count and run the script with --usage --confirm: the older row is removed and the yearly total of the title in erm_usage_yus is correct again Assisted-by: Sonnet 5.5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210276|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #41 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210417 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210417&action=edit Bug 41307: DBIC schema changes Generated for the new erm_counter_logs columns. The relationship between the counter log and its background job is added above the marker by the loader (belongs_to background_job, LEFT JOIN, on_delete SET NULL) and BackgroundJob gains has_many erm_counter_logs. Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210278|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #42 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210617 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210617&action=edit Bug 41307: Record the outcome of every SUSHI harvest in the counter logs Until now a counter log row was only written after a COUNTER file was stored, and logdetails was always empty. A harvest that failed left no record, and any exception in the SUSHI response blocked the import, including warnings such as 3040 Partial Data Returned. This patch writes one counter log row for every harvest attempt and for every file upload: - harvest_sushi collects the exceptions of the SUSHI response (top level exception, Exceptions array, Report_Header.Exceptions, vendor error bodies) and classifies the outcome as success, no_usage, no_data, retry or error. Code 3030 No Usage Available without items gives no_usage: the month is complete, the usage is zero. Code 3032, HTTP 204 and an empty report give no_data. Retry codes are 1000, 1010, 1011, 1020, 3031 and 3040, plus HTTP 202, 429 and 5xx. The request parameter warnings 3050, 3060, 3061 and 3062 are informational: the report is complete. A response that is not JSON is an error. - Reports with items are stored even when they carry a warning exception; the log is flagged retry for 3040 and 3031. A report with items and an error exception is not imported: it is logged as error. - logdetails holds a JSON document with the HTTP status, the exceptions (code, severity, message, data), an optional free text message and the COUNTER release. The job messages carry the exception message and data. The request URL is never stored or logged, it holds the API key. The HTTP failure is logged through Koha::Logger instead of warn. - A harvest without a file has a log row without a file name. One row per background job: a failure after the row was written does not add one. - CounterFile::store accepts the log fields of a harvest. For an uploaded file it reads the report type, the period and the Exceptions row from the file header. - CounterLog gains sushi_code_status, decoded_logdetails and a background_job accessor. - The background job passes its id to the harvest and reports the harvest status, shown on the job detail page with a badge per status. Test plan: 1. Run updatedatabase. Open a provider that had import logs before the patch, tab "Import logs": the old rows show status "Success" 2. Run a harvest over two months. On the Import logs tab, check one row per report type and month with the columns Filename, Report type, Period, Status, Details, Import date, Imported by and Job. The job link opens the background job page, which shows a "Harvest status" badge 3. Create a provider with wrong credentials and run a harvest: the job finishes, the Import logs tab shows a row with status "Error", no file name, the exception code and message in Details, and no Download or Delete action. Change the URL to a host that does not exist and run a harvest again: the new row shows status "Retry" with HTTP 500 and the connection error in Details 4. Harvest a month for which the provider has no usage (if the provider allows it): the row shows status "No usage" or "No data" and the job page shows the same badge 5. Use the Status filter of the table: select "Error", then "Success" and check the rows shown 6. Upload a COUNTER file on the Manual upload tab: its row shows the report type and the period read from the file header, status "Success", Download and Delete actions, and no job link 7. Delete the uploaded file: the row disappears 8. Age the harvest jobs so the cleanup can purge them, in koha-mysql: UPDATE background_jobs SET ended_on = DATE_SUB(NOW(), INTERVAL 2 DAY) WHERE type = 'erm_sushi_harvester'; Then run misc/cronjobs/cleanup_database.pl --jobs-days 1 --jobs-type erm_sushi_harvester --confirm. The import log rows stay, with an empty Job column, and the job link is gone 9. prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210281|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #43 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210618 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210618&action=edit Bug 41307: Unit tests CounterLog.t covers sushi_code_status, decoded_logdetails and the background_job accessor. UsageDataProvider.t traces every outcome of harvest_sushi: items with and without exceptions, the retry, no_usage, no_data and error codes, a request parameter warning next to items, an error exception next to items (not imported), HTTP failures, a response that is not JSON, a file too large and an unsupported release; the log row, its details and the job messages are checked for each. The harvester job tests check the status in the job report and the log row of the job. The older harvester test keeps its message format: message and data, without the severity. The API test covers the status filter and the background_job embed. The Cypress test covers the import logs tab. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Lisa G <lgonzalez@palni.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lgonzalez@palni.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210275|0 |1 is obsolete| | Attachment #210277|0 |1 is obsolete| | Attachment #210279|0 |1 is obsolete| | Attachment #210280|0 |1 is obsolete| | Attachment #210417|0 |1 is obsolete| | Attachment #210617|0 |1 is obsolete| | Attachment #210618|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #44 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210906 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210906&action=edit Bug 41307: Add status, background_job_id and report period to erm_counter_logs A counter log row was only written after a COUNTER file was stored, so a failed harvest left no trace and a log could not be tied to the job that produced it. This adds to erm_counter_logs: - background_job_id, a foreign key to background_jobs with ON DELETE SET NULL, so a log survives the purge of its job by cleanup_database - report_type, report_year and report_month, the harvested period of a single-month job, so the import logs can be read per month - status, the outcome of the harvest or import: success, no_usage, no_data, retry or error, with an index for the status filter Existing rows get status success: before this patch a row was only written after a successful import. The counter log API definition gains the same fields. Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #45 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210907 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210907&action=edit Bug 41307: DBIC schema changes Generated for the new erm_counter_logs columns. The relationship between the counter log and its background job is added above the marker by the loader (belongs_to background_job, LEFT JOIN, on_delete SET NULL) and BackgroundJob gains has_many erm_counter_logs. Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #46 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210908 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210908&action=edit Bug 41307: Tolerate COUNTER header exceptions without Data The CSV header builders for COUNTER 5 and 5.1 concatenated Code, Message and Data of every Report_Header exception. An exception without Data (or without Message) raised an "uninitialized value" warning. Reports that carry a warning exception such as 3040 Partial Data Returned are now stored, so the header must handle these fields when they are absent. Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #47 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210909 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210909&action=edit Bug 41307: Record the outcome of every SUSHI harvest in the counter logs Until now a counter log row was only written after a COUNTER file was stored, and logdetails was always empty. A harvest that failed left no record, and any exception in the SUSHI response blocked the import, including warnings such as 3040 Partial Data Returned. This patch writes one counter log row for every harvest attempt and for every file upload: - harvest_sushi collects the exceptions of the SUSHI response (top level exception, Exceptions array, Report_Header.Exceptions, vendor error bodies) and classifies the outcome as success, no_usage, no_data, retry or error. Code 3030 No Usage Available without items gives no_usage: the month is complete, the usage is zero. Code 3032, HTTP 204 and an empty report give no_data. Retry codes are 1000, 1010, 1011, 1020, 3031 and 3040, plus HTTP 202, 429 and 5xx. The request parameter warnings 3050, 3060, 3061 and 3062 are informational: the report is complete. A response that is not JSON is an error. - Reports with items are stored even when they carry a warning exception; the log is flagged retry for 3040 and 3031. A report with items and an error exception is not imported: it is logged as error. - logdetails holds a JSON document with the HTTP status, the exceptions (code, severity, message, data), an optional free text message and the COUNTER release. The job messages carry the exception message and data. The request URL is never stored or logged, it holds the API key. The HTTP failure is logged through Koha::Logger instead of warn. - A harvest without a file has a log row without a file name. One row per background job: a failure after the row was written does not add one. - CounterFile::store accepts the log fields of a harvest. For an uploaded file it reads the report type, the period and the Exceptions row from the file header. - CounterLog gains sushi_code_status, decoded_logdetails and a background_job accessor. - The background job passes its id to the harvest and reports the harvest status, shown on the job detail page with a badge per status. Test plan: 1. Run updatedatabase. Open a provider that had import logs before the patch, tab "Import logs": the old rows show status "Success" 2. Run a harvest over two months. On the Import logs tab, check one row per report type and month with the columns Filename, Report type, Period, Status, Details, Import date, Imported by and Job. The job link opens the background job page, which shows a "Harvest status" badge 3. Create a provider with wrong credentials and run a harvest: the job finishes, the Import logs tab shows a row with status "Error", no file name, the exception code and message in Details, and no Download or Delete action. Change the URL to a host that does not exist and run a harvest again: the new row shows status "Retry" with HTTP 500 and the connection error in Details 4. Harvest a month for which the provider has no usage (if the provider allows it): the row shows status "No usage" or "No data" and the job page shows the same badge 5. Use the Status filter of the table: select "Error", then "Success" and check the rows shown 6. Upload a COUNTER file on the Manual upload tab: its row shows the report type and the period read from the file header, status "Success", Download and Delete actions, and no job link 7. Delete the uploaded file: the row disappears 8. Age the harvest jobs so the cleanup can purge them, in koha-mysql: UPDATE background_jobs SET ended_on = DATE_SUB(NOW(), INTERVAL 2 DAY) WHERE type = 'erm_sushi_harvester'; Then run misc/cronjobs/cleanup_database.pl --jobs-days 1 --jobs-type erm_sushi_harvester --confirm. The import log rows stay, with an empty Job column, and the job link is gone 9. prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #48 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210910 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210910&action=edit Bug 41307: Expose status and the background job on the counter logs API The counter log definition gains status, and the list endpoint can embed background_job. Filtering on status already works through the q parameter, for example q={"status":"retry"}. Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #49 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210911 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210911&action=edit Bug 41307: Show harvest status in the import logs tab The Import logs tab of a usage data provider now shows the report type, the period, a status badge with a select filter, the exception details and a link to the background job. Download and delete are only offered for rows that have a counter file. The empty state counts counter logs instead of counter files, so failed harvests are listed too. Test plan: 1. Run a harvest for a provider over two months 2. Open the provider, Import logs tab 3. Check one row per report type and month, with status, period and a job link 4. Filter the status column on "Retry needed" 5. Upload a COUNTER file and check its row Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #50 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210912 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210912&action=edit Bug 41307: Unit tests CounterLog.t covers sushi_code_status, decoded_logdetails and the background_job accessor. UsageDataProvider.t traces every outcome of harvest_sushi: items with and without exceptions, the retry, no_usage, no_data and error codes, a request parameter warning next to items, an error exception next to items (not imported), HTTP failures, a response that is not JSON, a file too large and an unsupported release; the log row, its details and the job messages are checked for each. The harvester job tests check the status in the job report and the log row of the job. The older harvester test keeps its message format: message and data, without the severity. The API test covers the status filter and the background_job embed. The Cypress test covers the import logs tab. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #51 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210913 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210913&action=edit Bug 41307: Backfill the report type and period of the old counter logs The counter log rows written before Bug 41307 have no report type and no period, so the Import logs tab shows those rows without a report type or a period, and their months cannot be read one by one. The atomicupdate now backfills the rows that have no period, after it adds the columns: - the report type is the one of the counter file - the months come from the Reporting_Period header row of the file, or from its month columns when the file has no such row. Before Bug 41294 one harvest covered several months, so such a row becomes one row per month, all with the same file, import date, user and status - every month gets status success: the old code only wrote a log row after a file was imported - the background job is not linked A row whose file has no reporting period, or whose report type is longer than 10 characters, is reported and left without a period. Only the first 4000 bytes of each file are read. A row that already has a period is not read again, so running updatedatabase again changes nothing. Test plan: 1. On a database with harvests made before this patch, or after UPDATE erm_counter_logs SET report_type = NULL, report_year = NULL, report_month = NULL; and removing the new columns, run updatedatabase 2. Check that the rows now have a report type and a period, and that a file that covered several months has one row per month, all with the same file name 3. Open the provider, Import logs tab: every old row shows its report type and period, with the Download link 4. Run updatedatabase again, or run the update with the rows already backfilled: no counter log is changed Assisted-by: Sonnet 5.5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210282|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210283|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210906|0 |1 is obsolete| | Attachment #210907|0 |1 is obsolete| | Attachment #210908|0 |1 is obsolete| | Attachment #210909|0 |1 is obsolete| | Attachment #210910|0 |1 is obsolete| | Attachment #210911|0 |1 is obsolete| | Attachment #210912|0 |1 is obsolete| | Attachment #210913|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #52 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210945 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210945&action=edit Bug 41307: Add status, background_job_id and report period to erm_counter_logs A counter log row was only written after a COUNTER file was stored, so a failed harvest left no trace and a log could not be tied to the job that produced it. This adds to erm_counter_logs: - background_job_id, a foreign key to background_jobs with ON DELETE SET NULL, so a log survives the purge of its job by cleanup_database - report_type, report_year and report_month, the harvested period of a single-month job, so the import logs can be read per month - status, the outcome of the harvest or import: success, no_usage, no_data, retry or error, with an index for the status filter Existing rows get status success: before this patch a row was only written after a successful import. The counter log API definition gains the same fields. Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #53 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210946 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210946&action=edit Bug 41307: DBIC schema changes Generated for the new erm_counter_logs columns. The relationship between the counter log and its background job is added above the marker by the loader (belongs_to background_job, LEFT JOIN, on_delete SET NULL) and BackgroundJob gains has_many erm_counter_logs. Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #54 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210947 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210947&action=edit Bug 41307: Tolerate COUNTER header exceptions without Data The CSV header builders for COUNTER 5 and 5.1 concatenated Code, Message and Data of every Report_Header exception. An exception without Data (or without Message) raised an "uninitialized value" warning. Reports that carry a warning exception such as 3040 Partial Data Returned are now stored, so the header must handle these fields when they are absent. Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #55 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210948 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210948&action=edit Bug 41307: Record the outcome of every SUSHI harvest in the counter logs Until now a counter log row was only written after a COUNTER file was stored, and logdetails was always empty. A harvest that failed left no record, and any exception in the SUSHI response blocked the import, including warnings such as 3040 Partial Data Returned. This patch writes one counter log row for every harvest attempt and for every file upload: - harvest_sushi collects the exceptions of the SUSHI response (top level exception, Exceptions array, Report_Header.Exceptions, vendor error bodies) and classifies the outcome as success, no_usage, no_data, retry or error. Code 3030 No Usage Available without items gives no_usage: the month is complete, the usage is zero. Code 3032, HTTP 204 and an empty report give no_data. Retry codes are 1000, 1010, 1011, 1020, 3031 and 3040, plus HTTP 202, 429 and 5xx. The request parameter warnings 3050, 3060, 3061 and 3062 are informational: the report is complete. A response that is not JSON is an error. - Reports with items are stored even when they carry a warning exception; the log is flagged retry for 3040 and 3031. A report with items and an error exception is not imported: it is logged as error. - logdetails holds a JSON document with the HTTP status, the exceptions (code, severity, message, data), an optional free text message and the COUNTER release. The job messages carry the exception message and data. The request URL is never stored or logged, it holds the API key. The HTTP failure is logged through Koha::Logger instead of warn. - A harvest without a file has a log row without a file name. One row per background job: a failure after the row was written does not add one. - CounterFile::store accepts the log fields of a harvest. For an uploaded file it reads the report type, the period and the Exceptions row from the file header. - CounterLog gains sushi_code_status, decoded_logdetails and a background_job accessor. - The background job passes its id to the harvest and reports the harvest status, shown on the job detail page with a badge per status. Test plan: 1. Run updatedatabase. Open a provider that had import logs before the patch, tab "Import logs": the old rows show status "Success" 2. Run a harvest over two months. On the Import logs tab, check one row per report type and month with the columns Filename, Report type, Period, Status, Details, Import date, Imported by and Job. The job link opens the background job page, which shows a "Harvest status" badge 3. Create a provider with wrong credentials and run a harvest: the job finishes, the Import logs tab shows a row with status "Error", no file name, the exception code and message in Details, and no Download or Delete action. Change the URL to a host that does not exist and run a harvest again: the new row shows status "Retry" with HTTP 500 and the connection error in Details 4. Harvest a month for which the provider has no usage (if the provider allows it): the row shows status "No usage" or "No data" and the job page shows the same badge 5. Use the Status filter of the table: select "Error", then "Success" and check the rows shown 6. Upload a COUNTER file on the Manual upload tab: its row shows the report type and the period read from the file header, status "Success", Download and Delete actions, and no job link 7. Delete the uploaded file: the row disappears 8. Age the harvest jobs so the cleanup can purge them, in koha-mysql: UPDATE background_jobs SET ended_on = DATE_SUB(NOW(), INTERVAL 2 DAY) WHERE type = 'erm_sushi_harvester'; Then run misc/cronjobs/cleanup_database.pl --jobs-days 1 --jobs-type erm_sushi_harvester --confirm. The import log rows stay, with an empty Job column, and the job link is gone 9. prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #56 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210949 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210949&action=edit Bug 41307: Expose status and the background job on the counter logs API The counter log definition gains status, and the list endpoint can embed background_job. Filtering on status already works through the q parameter, for example q={"status":"retry"}. Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #57 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210950 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210950&action=edit Bug 41307: Show harvest status in the import logs tab The Import logs tab of a usage data provider now shows the report type, the period, a status badge with a select filter, the exception details and a link to the background job. Download and delete are only offered for rows that have a counter file. The empty state counts counter logs instead of counter files, so failed harvests are listed too. Test plan: 1. Run a harvest for a provider over two months 2. Open the provider, Import logs tab 3. Check one row per report type and month, with status, period and a job link 4. Filter the status column on "Retry needed" 5. Upload a COUNTER file and check its row Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #58 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210951 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210951&action=edit Bug 41307: Unit tests CounterLog.t covers sushi_code_status, decoded_logdetails and the background_job accessor. UsageDataProvider.t traces every outcome of harvest_sushi: items with and without exceptions, the retry, no_usage, no_data and error codes, a request parameter warning next to items, an error exception next to items (not imported), HTTP failures, a response that is not JSON, a file too large and an unsupported release; the log row, its details and the job messages are checked for each. The harvester job tests check the status in the job report and the log row of the job. The older harvester test keeps its message format: message and data, without the severity. The API test covers the status filter and the background_job embed. The Cypress test covers the import logs tab. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #59 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210952 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210952&action=edit Bug 41307: Backfill the report type and period of the old counter logs The counter log rows written before Bug 41307 have no report type and no period, so the Import logs tab shows those rows without a report type or a period, and their months cannot be read one by one. The atomicupdate now backfills the rows that have no period, after it adds the columns: - the report type is the one of the counter file - the months come from the Reporting_Period header row of the file, or from its month columns when the file has no such row. Before Bug 41294 one harvest covered several months, so such a row becomes one row per month, all with the same file, import date, user and status - every month gets status success: the old code only wrote a log row after a file was imported - the background job is not linked A row whose file has no reporting period, or whose report type is longer than 10 characters, is reported and left without a period. Only the first 4000 bytes of each file are read. A row that already has a period is not read again, so running updatedatabase again changes nothing. Test plan: 1. On a database with harvests made before this patch, or after UPDATE erm_counter_logs SET report_type = NULL, report_year = NULL, report_month = NULL; and removing the new columns, run updatedatabase 2. Check that the rows now have a report type and a period, and that a file that covered several months has one row per month, all with the same file name 3. Open the provider, Import logs tab: every old row shows its report type and period, with the Download link 4. Run updatedatabase again, or run the update with the rows already backfilled: no counter log is changed Assisted-by: Sonnet 5.5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 --- Comment #60 from Michaela Sieber <michaela.sieber@kit.edu> --- Signed off using a sandbox -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |nick@bywatersolutions.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=43526 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org