[Bug 38822] New: View easilty the months for which usage reports have been collected for a given source
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Bug ID: 38822 Summary: View easilty the months for which usage reports have been collected for a given source Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: ERM Assignee: jonathan.druart@gmail.com Reporter: mathsabypro@gmail.com QA Contact: testopia@bugs.koha-community.org CC: jonathan.druart@gmail.com, jonathan.field@ptfs-europe.com, martin.renvoize@ptfs-europe.com, matt.blenkinsop@ptfs-europe.com, pedro.amorim@ptfs-europe.com It could be nice to make it easy to view the months for which various usage reports have been collected for a given source (without the need to create a report in Koha) (FOLIO ERM or Coral could be source of inspiration) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #1 from Jonathan Field <jonathan.field@ptfs-europe.com> --- (In reply to Mathieu Saby from comment #0)
It could be nice to make it easy to view the months for which various usage reports have been collected for a given source (without the need to create a report in Koha) (FOLIO ERM or Coral could be source of inspiration)
Hi Mathieu, does the Data Providers Summary not give you what you need for this? URL is cgi-bin/koha/erm/eusage/usage_data_providers/summary. It was designed to replicate that Coral functionality. Jonathan -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Mathieu Saby <mathsabypro@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|View easilty the months for |View easily the months for |which usage reports have |which usage reports have |been collected for a given |been collected for a given |source |source -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #2 from Mathieu Saby <mathsabypro@gmail.com> --- Yes! I did not click on that link, because I thought I was already seeing a "summary" on the main page eusage/usage_data_providers So I agree the function exists. But it could be improved : - we only have a broad level of information : "title report", but it could be useful to display the harvesting status for specific reports (TR_B1, etc). - it is not possible to see "gaps" in harvesting, since we have only the 1st month and the last month. So I change the title of this bug -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Mathieu Saby <mathsabypro@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|View easily the months for |Make data provider summary |which usage reports have |more complete |been collected for a given | |source | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Mathieu Saby <mathsabypro@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Make data provider summary |Make data provider summary |more complete |more complete (all reports | |and all months) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew.auld@openfifth.co.uk | |, clemens.tubach@kit.edu, | |katrin.fischer@bsz-bw.de, | |michaela.sieber@kit.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #3 from Michaela Sieber <michaela.sieber@kit.edu> --- +1 We discussed that in Marseille this year. We (KIT) have a plugin solution for that and will try to find a way to improve Koha's core functionality. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Make data provider summary |ERM Usage data provider |more complete (all reports |summary should be more |and all months) |complete (all reports and | |all months) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #4 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 205792 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205792&action=edit Harvesting-Overview-monthly-Summery-Example-Plugin-KIT -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Sponsorship status|--- |Seeking cosponsors Comma delimited| |Karlsruhe Institute of list of Sponsors| |Technology (KIT) --- Comment #5 from Michaela Sieber <michaela.sieber@kit.edu> --- I have attached a Screenshot from our monthly harvesting overview which is part of our Plugin (https://gitlab.com/kit-library/koha-plugin-erm-harvesting-management) in the moment we do not have sufficient tracking of the harvesting jobs, that's why we needed to create a new database table in the plugin to create this overview. And we would need one harvesting job per month and not for periods. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jvinke@austinseminary.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #6 from John Vinke <jvinke@austinseminary.edu> --- +1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #7 from John Vinke <jvinke@austinseminary.edu> --- One thing that would be very helpful is what’s shown in Michaela’s screenshot. There’s a success message for months where the harvest was successful but there was no usage. Right now, it’s not obvious whether the harvest failed or whether there just wasn’t any usage for that month. A string like "No Usage" or recording uses as 0 would help immensely. The way this is currently implemented compounds some of the other pain points with COUNTER harvests. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41305 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|jonathan.druart@gmail.com |matt.blenkinsop@openfifth.c | |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Comma delimited|Karlsruhe Institute of |Karlsruhe Institute of list of Sponsors|Technology (KIT) |Technology (KIT), OpenFitfh Priority|P5 - low |P2 Initiative type|--- |Epic Sponsorship status|Seeking cosponsors |Sponsored Status|NEW |ASSIGNED Strategic theme|--- |User Experience -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |26.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |43904 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43904 [Bug 43904] [OMNIBUS] ERM Usage Improvements -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|43904 | Blocks| |43904 --- Comment #8 from Michaela Sieber <michaela.sieber@kit.edu> --- Link to Omnibus Bug to track all ERM Usage Improvements Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43904 [Bug 43904] [OMNIBUS] ERM Usage Improvements -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42088 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42088 [Bug 42088] ERM usage data provider summary not usable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |41305 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41305 [Bug 41305] Make the list of harvesting jobs more informative in erm/eusage/usage_data_providers -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #9 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 208977 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208977&action=edit 38822_Screenshot_new-Provider-Summery -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |christian.nelson@uwasa.fi, | |johanna.miettunen@haaga-hel | |ia.fi, | |veli-matti.hakkinen@jyu.fi -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |41307 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307 [Bug 41307] Improve erm_counter_logs and automatically indicate when a harvest job is completed (successfully or with an error) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209183|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #16 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 209224 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209224&action=edit Bug 38822: Add a period index and the erm_counter_log_summary view The data providers summary page needs one row per usage data provider, report type and harvest year, with the status of the latest counter log for each month. This patch adds the database side of that: - An index on erm_counter_logs (usage_data_provider_id, report_type, report_year, report_month), the natural key of a harvested month. It serves the view below, the Import logs tab and the retry lookup. - A view erm_counter_log_summary, in kohastructure.sql next to erm_usage_yus. It keeps the newest log of each month (by importdate, then id) with a window function and pivots the twelve statuses into month_1 to month_12. Logs without a single-month period (multi-month harvests, uploads covering several months) are not included. MIN(erm_counter_log_id) gives the row a stable primary key for DBIC. The derived table is materialised on every query, so the view scans the single-month logs each time. With the index this stays in milliseconds for tens of thousands of logs. updatedatabase runs the atomicupdate files in name order, so this file runs before bug_41307.pl, which adds the report_type, report_year, report_month and status columns the index and the view read. The file adds them too, under the same guards, so the first run of updatedatabase works in either order and the file that runs second skips them. Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |41306 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41306 [Bug 41306] ERM Usage Date picker for COUNTER data harvesting should not ask to choose a day -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209184|0 |1 is obsolete| | Attachment #209185|0 |1 is obsolete| | Attachment #209186|0 |1 is obsolete| | Attachment #209187|0 |1 is obsolete| | Attachment #209188|0 |1 is obsolete| | Attachment #209224|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #17 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209373 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209373&action=edit Bug 38822: DBIC files for erm_counter_log_summary Generated with the view loaded into the dbic database. The primary key and the relationship to the usage data provider are added by hand below the marker, as for erm_usage_yus. Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #18 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209374 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209374&action=edit Bug 38822: Add a counter log summary API GET /erm/counter_logs/summary lists the erm_counter_log_summary view through the read-only Koha::ERM::EUsage::CounterLogSummary class, with the usual q, _order_by and paging parameters. Searching and ordering on provider_name needs no embed because the view carries the name. Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #19 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209375 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209375&action=edit Bug 38822: Rebuild the data providers summary from the counter logs The summary page showed the earliest and latest usage month per report family, which cost twelve aggregate queries per provider plus the content of every counter file embedded in the response. It now shows one row per harvest year, provider and report type with a status badge for each month, read from the counter log summary API in one request. The provider name opens the provider page. The first three columns can be sorted and filtered; the default order is year descending, then provider and report type. The status labels and badge classes move from the import logs tab to a shared module so both tables render the same badges. The retry badge reads "Retry" on the import logs tab, the summary page and the job page; it read "Retry needed" before. A failed count request no longer leaves the summary page on "Loading". Test plan: 1. With no harvest logs, open eUsage > Data providers > "Data providers summary": the page shows "There are no harvest logs to summarize" 2. Harvest three months for one provider, then open the summary page: one row for the provider, the report type and the harvest year, with a status badge under each harvested month and nothing under the other months. Hover a badge: the tooltip shows the status 3. Harvest a month of a different year and a month of a second report type: the page shows one row per year and per report type, ordered by year descending, then provider, then report type 4. Sort and filter the "Harvest year", "Provider" and "Report type" columns 5. Click the provider name: the provider page opens 6. Run a failing harvest (wrong credentials): the summary shows an "Error" badge for that month. Harvest the month again with correct credentials: the badge changes to "Success", only the latest log of a month counts 7. Check that a badge with the status "Retry" reads "Retry" on the summary page, the Import logs tab and the background job page 8. Edit a provider and save it without changes: no error (the removed earliest/latest fields of the API are no longer sent) 9. The data providers list still shows the "Last run" column 10. prove -l t/db_dependent/Koha/ERM/EUsage/CounterLogSummary.t \ t/db_dependent/api/v1/erm_counter_log_summaries.t \ t/db_dependent/api/v1/erm_usage_data_providers.t 11. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #20 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209376 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209376&action=edit Bug 38822: Remove the per-provider date aggregates from the providers API The earliest_* and latest_* fields were only used by the old summary page. Computing them ran twelve aggregate queries on erm_usage_mus for every provider in the list. last_run stays, the providers list shows it. API change: the earliest_* and latest_* fields are removed from the erm_usage_data_provider definition. additionalProperties is false, so a client that reads a provider and sends these fields back now gets a 400. The release notes should mention it. Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #21 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209377 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209377&action=edit Bug 38822: Unit tests for the counter log summary Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/CounterLogSummary.t \ t/db_dependent/api/v1/erm_counter_log_summaries.t \ t/db_dependent/api/v1/erm_usage_data_providers.t yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38822 --- Comment #22 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209378 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209378&action=edit Bug 38822: Add a period index and the erm_counter_log_summary view The data providers summary page needs one row per usage data provider, report type and harvest year, with the status of the latest counter log for each month. This patch adds the database side of that: - An index on erm_counter_logs (usage_data_provider_id, report_type, report_year, report_month), the natural key of a harvested month. It serves the view below, the Import logs tab and the retry lookup. - A view erm_counter_log_summary, in kohastructure.sql next to erm_usage_yus. It keeps the newest log of each month (by importdate, then id) with a window function and pivots the twelve statuses into month_1 to month_12. Logs without a single-month period (multi-month harvests, uploads covering several months) are not included. MIN(erm_counter_log_id) gives the row a stable primary key for DBIC. The derived table is materialised on every query, so the view scans the single-month logs each time. With the index this stays in milliseconds for tens of thousands of logs. updatedatabase runs the atomicupdate files in name order, so this file runs before bug_41307.pl, which adds the report_type, report_year, report_month and status columns the index and the view read. The file adds them too, under the same guards, so the first run of updatedatabase works in either order and the file that runs second skips them. Sponsored-by: Karlsruhe Institute of Technology (KIT) Assisted-by: Fable 5.1 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org