[Bug 42218] New: ERM Usage – remove Database table erm_usage_yus ?
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Bug ID: 42218 Summary: ERM Usage – remove Database table erm_usage_yus ? Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: ERM Assignee: jonathan.druart@gmail.com Reporter: michaela.sieber@kit.edu QA Contact: testopia@bugs.koha-community.org CC: jonathan.druart@gmail.com, jonathan.field@openfifth.co.uk, martin.renvoize@openfifth.co.uk, matt.blenkinsop@openfifth.co.uk, pedro.amorim@openfifth.co.uk The erm_usage_yus table could be removed completely, as it contains redundant information which already exists in the erm_usage_mus -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=42215 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|jonathan.druart@gmail.com |matt.blenkinsop@openfifth.c | |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43904 Status|NEW |ASSIGNED Target Milestone|--- |26.11 --- Comment #1 from Michaela Sieber <michaela.sieber@kit.edu> --- Link to Omnibus Bug to track all ERM Usage Improvements Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43904 [Bug 43904] [OMNIBUS] ERM Usage Improvements -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42214 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42214 [Bug 42214] ERM usage database table erm_usage_yus contains incorrect data -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42215 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42215 [Bug 42215] ERM Usage Reports (Yearly) – Incorrect Data -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42217 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42217 [Bug 42217] ERM Usage – Harvesting Twice Does Not Overwrite Data -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #2 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 209113 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209113&action=edit Bug 42218: Create a shared UsageObject class The four usage classes (UsageTitle, UsagePlatform, UsageDatabase and UsageItem) had byte-identical monthly_usages, yearly_usages and erm_usage_*uses methods. This moves them to one base class, Koha::ERM::EUsage::UsageObject, so the yearly usage change in the next patches is made once. Each subclass keeps only _type. Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #3 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 209114 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209114&action=edit Bug 42218: Remove erm_usage_yus references from the code Yearly usage rows were written by CounterFile per COUNTER file: the months of one file were summed and the row was deduplicated on a hash that included the total. A re-harvest of the same year with a different total added a second row, and a year spread over two files gave two partial totals. The yearly numbers were wrong and could not be repaired. The next patches replace the erm_usage_yus table with a view that sums erm_usage_mus per year. This patch removes every writer first so the switch is safe: - CounterFile no longer builds or stores yearly usage entries - UsageObject->yearly_usages is a plain getter, the same as erm_usage_yuses - Koha::ERM::EUsage::YearlyUsage->store and ->delete throw, the object is read-only - the SUSHI harvester job report drops the added_yus and skipped_yus counters and the template no longer shows them Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #4 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 209115 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209115&action=edit Bug 42218: Remove erm_usage_yus and replace with an SQL view erm_usage_yus becomes a view that sums erm_usage_mus per year, grouped on every key column (usage object, provider, year, metric type, access type, yop and report type). A re-harvest updates the monthly rows and the yearly totals follow; rows cannot duplicate. Dropping the table loses no data: every row in it was written from the non-zero monthly rows of the same COUNTER file, so the view rebuilds the same totals, corrected where files overlapped. The view is defined in kohastructure.sql next to the tables it reads, so a fresh install, the KTD dbic helper and misc4dev all create it. The atomicupdate drops the table when it is still a base table and creates the view with CREATE OR REPLACE, so it can run more than once. MIN(monthly_usage_summary_id) gives each row a stable key for DBIC. Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #5 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 209116 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209116&action=edit Bug 42218: DBIC files Regenerated with the view in place. The schema loader marks ErmUsageYus as a DBIx::Class::ResultSource::View and gives a view no primary key and no relationships, so these are added by hand below the marker: - the primary key and is_nullable => 0 on yearly_usage_summary_id in ErmUsageYus.pm, with the belongs_to relations to the usage objects and the provider - the has_many erm_usage_yuses relation in ErmUsageTitle, ErmUsagePlatform, ErmUsageDatabase, ErmUsageItem and ErmUsageDataProvider Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #6 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 209117 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209117&action=edit Bug 42218: Update test builder to recognise SQL views TestBuilder inserts rows, which is not possible on a view. build() now carps and returns undef when the source is a DBIx::Class::ResultSource::View, and the build_object() sweep over every Koha::Object class skips the classes backed by a view, so Koha::ERM::EUsage::YearlyUsage does not break TestBuilder.t. Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #7 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 209118 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209118&action=edit Bug 42218: Update unit tests The harvester and CounterFile tests no longer count added or skipped yearly rows. They check instead that the erm_usage_yus view holds the summed totals, and that a second harvest of the same file leaves the yearly count unchanged. YearlyUsage.t covers the view per year and per metric type, the UsageObject getter, to_api, and the read-only store and delete. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/YearlyUsage.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/TestBuilder.t yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209113|0 |1 is obsolete| | Attachment #209114|0 |1 is obsolete| | Attachment #209115|0 |1 is obsolete| | Attachment #209116|0 |1 is obsolete| | Attachment #209117|0 |1 is obsolete| | Attachment #209118|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #8 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209149 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209149&action=edit Bug 42218: Create a shared UsageObject class The four usage classes (UsageTitle, UsagePlatform, UsageDatabase and UsageItem) had byte-identical monthly_usages, yearly_usages and erm_usage_*uses methods. This moves them to one base class, Koha::ERM::EUsage::UsageObject, so the yearly usage change in the next patches is made once. Each subclass keeps only _type. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #9 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209150 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209150&action=edit Bug 42218: Remove erm_usage_yus references from the code Yearly usage rows were written by CounterFile per COUNTER file: the months of one file were summed and the row was deduplicated on a hash that included the total. A re-harvest of the same year with a different total added a second row, and a year spread over two files gave two partial totals. The yearly numbers were wrong and could not be repaired. The next patches replace the erm_usage_yus table with a view that sums erm_usage_mus per year. This patch removes every writer first so the switch is safe: - CounterFile no longer builds or stores yearly usage entries - UsageObject->yearly_usages is a plain getter, the same as erm_usage_yuses - Koha::ERM::EUsage::YearlyUsage->store and ->delete throw, the object is read-only - the SUSHI harvester job report drops the added_yus and skipped_yus counters and the template no longer shows them Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #10 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209151 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209151&action=edit Bug 42218: Remove erm_usage_yus and replace with an SQL view erm_usage_yus becomes a view that sums erm_usage_mus per year, grouped on every key column (usage object, provider, year, metric type, access type, yop and report type). A re-harvest updates the monthly rows and the yearly totals follow; rows cannot duplicate. Dropping the table loses no data: every row in it was written from the non-zero monthly rows of the same COUNTER file, so the view rebuilds the same totals, corrected where files overlapped. The view is defined in kohastructure.sql next to the tables it reads, so a fresh install, the KTD dbic helper and misc4dev all create it. The atomicupdate drops the table when it is still a base table and creates the view with CREATE OR REPLACE, so it can run more than once. MIN(monthly_usage_summary_id) gives each row a stable key for DBIC. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #11 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209152 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209152&action=edit Bug 42218: DBIC files Regenerated with the view in place. The schema loader marks ErmUsageYus as a DBIx::Class::ResultSource::View and gives a view no primary key and no relationships, so these are added by hand below the marker: - the primary key and is_nullable => 0 on yearly_usage_summary_id in ErmUsageYus.pm, with the belongs_to relations to the usage objects and the provider - the has_many erm_usage_yuses relation in ErmUsageTitle, ErmUsagePlatform, ErmUsageDatabase, ErmUsageItem and ErmUsageDataProvider Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #12 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209153 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209153&action=edit Bug 42218: Update test builder to recognise SQL views TestBuilder inserts rows, which is not possible on a view. build() now carps and returns undef when the source is a DBIx::Class::ResultSource::View, and the build_object() sweep over every Koha::Object class skips the classes backed by a view, so Koha::ERM::EUsage::YearlyUsage does not break TestBuilder.t. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #13 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 209154 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209154&action=edit Bug 42218: Update unit tests The harvester and CounterFile tests no longer count added or skipped yearly rows. They check instead that the erm_usage_yus view holds the summed totals, and that a second harvest of the same file leaves the yearly count unchanged. YearlyUsage.t covers the view per year and per metric type, the UsageObject getter, to_api, and the read-only store and delete. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/YearlyUsage.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/TestBuilder.t yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |critical --- Comment #14 from Michaela Sieber <michaela.sieber@kit.edu> --- I am changing the importance from "Enhancement" to "Critical Bug", as this is the only reasonable solution to the bug described in detail in Bug 42214, as also reflected in the commit messages of the attached patches. At the same time, this also fixes the bug described in 42215, since the table containing the incorrect data will no longer exist. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_26_11_candidate CC| |christian.nelson@uwasa.fi, | |clemens.tubach@kit.edu, | |jvinke@austinseminary.edu, | |katrin.fischer@bsz-bw.de -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |nick@bywatersolutions.com |y.org | Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209149|0 |1 is obsolete| | Attachment #209150|0 |1 is obsolete| | Attachment #209151|0 |1 is obsolete| | Attachment #209152|0 |1 is obsolete| | Attachment #209153|0 |1 is obsolete| | Attachment #209154|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #15 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 209998 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209998&action=edit Bug 42218: Create a shared UsageObject class The four usage classes (UsageTitle, UsagePlatform, UsageDatabase and UsageItem) had byte-identical monthly_usages, yearly_usages and erm_usage_*uses methods. This moves them to one base class, Koha::ERM::EUsage::UsageObject, so the yearly usage change in the next patches is made once. Each subclass keeps only _type. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #16 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 209999 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209999&action=edit Bug 42218: Remove erm_usage_yus references from the code Yearly usage rows were written by CounterFile per COUNTER file: the months of one file were summed and the row was deduplicated on a hash that included the total. A re-harvest of the same year with a different total added a second row, and a year spread over two files gave two partial totals. The yearly numbers were wrong and could not be repaired. The next patches replace the erm_usage_yus table with a view that sums erm_usage_mus per year. This patch removes every writer first so the switch is safe: - CounterFile no longer builds or stores yearly usage entries - UsageObject->yearly_usages is a plain getter, the same as erm_usage_yuses - Koha::ERM::EUsage::YearlyUsage->store and ->delete throw, the object is read-only - the SUSHI harvester job report drops the added_yus and skipped_yus counters and the template no longer shows them Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #17 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 210000 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210000&action=edit Bug 42218: Remove erm_usage_yus and replace with an SQL view erm_usage_yus becomes a view that sums erm_usage_mus per year, grouped on every key column (usage object, provider, year, metric type, access type, yop and report type). A re-harvest updates the monthly rows and the yearly totals follow; rows cannot duplicate. Dropping the table loses no data: every row in it was written from the non-zero monthly rows of the same COUNTER file, so the view rebuilds the same totals, corrected where files overlapped. The view is defined in kohastructure.sql next to the tables it reads, so a fresh install, the KTD dbic helper and misc4dev all create it. The atomicupdate drops the table when it is still a base table and creates the view with CREATE OR REPLACE, so it can run more than once. MIN(monthly_usage_summary_id) gives each row a stable key for DBIC. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #18 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 210001 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210001&action=edit Bug 42218: DBIC files Regenerated with the view in place. The schema loader marks ErmUsageYus as a DBIx::Class::ResultSource::View and gives a view no primary key and no relationships, so these are added by hand below the marker: - the primary key and is_nullable => 0 on yearly_usage_summary_id in ErmUsageYus.pm, with the belongs_to relations to the usage objects and the provider - the has_many erm_usage_yuses relation in ErmUsageTitle, ErmUsagePlatform, ErmUsageDatabase, ErmUsageItem and ErmUsageDataProvider Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #19 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 210002 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210002&action=edit Bug 42218: Update test builder to recognise SQL views TestBuilder inserts rows, which is not possible on a view. build() now carps and returns undef when the source is a DBIx::Class::ResultSource::View, and the build_object() sweep over every Koha::Object class skips the classes backed by a view, so Koha::ERM::EUsage::YearlyUsage does not break TestBuilder.t. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #20 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 210003 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210003&action=edit Bug 42218: Update unit tests The harvester and CounterFile tests no longer count added or skipped yearly rows. They check instead that the erm_usage_yus view holds the summed totals, and that a second harvest of the same file leaves the yearly count unchanged. YearlyUsage.t covers the view per year and per metric type, the UsageObject getter, to_api, and the read-only store and delete. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/YearlyUsage.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/TestBuilder.t yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #21 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Code reads well, nice simplification. I didn't have real world data for testing, but generated monthly usages and confirmed they are combined in the view when all the GROUP BY columns are matched. This looks to be the first official VIEW in the code, not sure if we need extra eyes on that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |The yearly numbers were release notes| |wrong and could not be | |repaired. | |The patches replace the | |erm_usage_yus table with a | |view that calculates the | |yearly totals by summing up | |the data from | |erm_usage_mus. | |Dropping the table does not | |result in any data loss. | |The erm_usage_mus View | |remains available for | |reporting, both in the ERM | |module and in the Reports | |module. Summary|ERM Usage – remove Database |ERM Usage – replace |table erm_usage_yus ? |Database Table | |erm_usage_yus with a View -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #22 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- I considered putting the view in a separate SQL file that future views could be added to but that causes issues with misc4dev and installations - open to exploring it further if more views are added -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |43944 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43944 [Bug 43944] Manual Counter file imports fail on spec compliant files -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Failed QA --- Comment #23 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Hi all, I don't think we should introduce a VIEW paradigm here (unless we really must): - It would be Koha's first, bringing new patterns and overhead: a DBIC view class and a TestBuilder special case. - Restoring a dump under another DB user breaks it. mysqldump carries the DEFINER (koha_kohadev@%), so a prod copy into test, a dump into KTD, or a server move makes every query on erm_usage_yus fail with ERROR 1449 (SQL SECURITY INVOKER would avoid this). - It becomes invalid in the DB if an erm_usage_mus column is later renamed or dropped: the ALTER would succeed but the next report execution would fail. - It re-sums all monthly usage every time a report runs. My suggestion going forward: - Keep patches 1 and 2 (with the view wording in 2 adjusted) and the erm_usage_yus table. - At the end of CounterFile->store, rebuild the yearly rows for the provider, report type and years just imported: in one transaction, delete them and insert a SUM over all of erm_usage_mus (same GROUP BY as the view). Putting this in a single method (e.g. Koha::ERM::EUsage::YearlyUsages->rebuild) keeps it reusable if anything else ever writes monthly usage. - Do the same over the whole table in the atomicupdate (plain SQL) to fix existing data. - Add a test checking that every erm_usage_mus column is grouped, summed or deliberately skipped, so a new column can't be silently missed. I believe this would fix 42214 and 42215 at once: patch 2 in this series already removes the duplicate check that skipped rows, and the rebuild I'm proposing above leaves a single row per title/year/metric (and access type or YOP where the report has them) holding the full total, rather than partial rows from whatever import just happened. yus stays as stored data, but since it's always rebuilt from mus, it can't drift as long as monthly writes go through the rebuild. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mark@hofstetter.at, | |raphael.straub@kit.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209998|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210003|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #24 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210267 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210267&action=edit Bug 42218: Create a shared UsageObject class The four usage classes (UsageTitle, UsagePlatform, UsageDatabase and UsageItem) had byte-identical monthly_usages, yearly_usages and erm_usage_*uses methods. This moves them to one base class, Koha::ERM::EUsage::UsageObject, so the yearly usage change in the next patches is made once. Each subclass keeps only _type. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #25 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210268 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210268&action=edit Bug 42218: Rebuild erm_usage_yus from the monthly usage Yearly usage rows were written by CounterFile per COUNTER file: the months of one file were summed and the row was deduplicated on a hash that included the total. A re-harvest of the same year with a different total added a second row, and a year spread over two files gave two partial totals. The yearly numbers were wrong and could not be repaired. erm_usage_yus stays a table, but its rows are now always the sum of erm_usage_mus per year, so erm_usage_mus is the only source of the yearly numbers: - YearlyUsages->rebuild deletes and inserts the rows of one provider, report type and year from the sum of erm_usage_mus over every key column (usage object, provider, year, metric type, access type, yop and report type). It locks the provider row, as the jobs of the months of one provider run in parallel - CounterFile no longer builds or stores yearly usage entries per file. When all rows are done it calls rebuild for each year of the file - UsageObject->yearly_usages is a plain getter, the same as erm_usage_yuses - Koha::ERM::EUsage::YearlyUsage->store and ->delete throw, the object is read-only - the SUSHI harvester job report drops the added_yus and skipped_yus counters and the template no longer shows them Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #26 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210269 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210269&action=edit Bug 42218: Update unit tests The harvester and CounterFile tests no longer count added or skipped yearly rows. They check instead that erm_usage_yus holds the summed totals after an import, and that a second harvest of the same file leaves the yearly count unchanged. YearlyUsage.t covers rebuild: the sum per year and per metric type, a changed or removed month, rows with NULL key columns, other years and report types, the UsageObject getter, to_api, and the read-only store and delete. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/YearlyUsage.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #209999|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210000|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210001|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210002|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210267|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210268|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210269|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #27 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210270 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210270&action=edit Bug 42218: Create a shared UsageObject class The four usage classes (UsageTitle, UsagePlatform, UsageDatabase and UsageItem) had byte-identical monthly_usages, yearly_usages and erm_usage_*uses methods. This moves them to one base class, Koha::ERM::EUsage::UsageObject, so the yearly usage change in the next patches is made once. Each subclass keeps only _type. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #28 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210271 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210271&action=edit Bug 42218: Rebuild erm_usage_yus from the monthly usage Yearly usage rows were written by CounterFile per COUNTER file: the months of one file were summed and the row was deduplicated on a hash that included the total. A re-harvest of the same year with a different total added a second row, and a year spread over two files gave two partial totals. The yearly numbers were wrong and could not be repaired. erm_usage_yus stays a table, but its rows are now always the sum of erm_usage_mus per year, so erm_usage_mus is the only source of the yearly numbers: - YearlyUsages->rebuild deletes and inserts the rows of one provider, report type and year from the sum of erm_usage_mus over every key column (usage object, provider, year, metric type, access type, yop and report type). It locks the provider row, as the jobs of the months of one provider run in parallel - CounterFile no longer builds or stores yearly usage entries per file. When all rows are done it calls rebuild for each year of the file - UsageObject->yearly_usages is a plain getter, the same as erm_usage_yuses - Koha::ERM::EUsage::YearlyUsage->store and ->delete throw, the object is read-only - the SUSHI harvester job report drops the added_yus and skipped_yus counters and the template no longer shows them Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #29 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210272 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210272&action=edit Bug 42218: Update unit tests The harvester and CounterFile tests no longer count added or skipped yearly rows. They check instead that erm_usage_yus holds the summed totals after an import, and that a second harvest of the same file leaves the yearly count unchanged. YearlyUsage.t covers rebuild: the sum per year and per metric type, a changed or removed month, rows with NULL key columns, other years and report types, the UsageObject getter, to_api, and the read-only store and delete. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/YearlyUsage.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Assisted-by: Sonnet 5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210270|0 |1 is obsolete| | Attachment #210271|0 |1 is obsolete| | Attachment #210272|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #30 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210311 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210311&action=edit Bug 42218: Create a shared UsageObject class The four usage classes (UsageTitle, UsagePlatform, UsageDatabase and UsageItem) had byte-identical monthly_usages, yearly_usages and erm_usage_*uses methods. This moves them to one base class, Koha::ERM::EUsage::UsageObject, so the yearly usage change in the next patches is made once. Each subclass keeps only _type. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #31 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210312 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210312&action=edit Bug 42218: Rebuild erm_usage_yus from the monthly usage Yearly usage rows were written by CounterFile per COUNTER file: the months of one file were summed and the row was deduplicated on a hash that included the total. A re-harvest of the same year with a different total added a second row, and a year spread over two files gave two partial totals. The yearly numbers were wrong and could not be repaired. erm_usage_yus stays a table, but its rows are now always the sum of erm_usage_mus per year, so erm_usage_mus is the only source of the yearly numbers: - YearlyUsages->rebuild deletes and inserts the rows of one provider, report type and year from the sum of erm_usage_mus over every key column (usage object, provider, year, metric type, access type, yop and report type). It locks the provider row, as the jobs of the months of one provider run in parallel - CounterFile no longer builds or stores yearly usage entries per file. When all rows are done it calls rebuild for each year of the file - UsageObject->yearly_usages is a plain getter, the same as erm_usage_yuses - Koha::ERM::EUsage::YearlyUsage->store and ->delete throw, the object is read-only - the SUSHI harvester job report drops the added_yus and skipped_yus counters and the template no longer shows them Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #32 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 210313 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210313&action=edit Bug 42218: Update unit tests The harvester and CounterFile tests no longer count added or skipped yearly rows. They check instead that erm_usage_yus holds the summed totals after an import, and that a second harvest of the same file leaves the yearly count unchanged. YearlyUsage.t covers rebuild: the sum per year and per metric type, a changed or removed month, rows with NULL key columns, other years and report types, the UsageObject getter, to_api, and the read-only store and delete. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/YearlyUsage.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #33 from Mark Hofstetter <mark@hofstetter.at> --- being asked for my opinion here, without really understanding the underlying problem: I am not a huge fan of views (esp aggregate) in sql for code because they obfuscate the perceived data structure. If you need some data why not simply excute the SQL itself, maybe nicely structured in a CTE (ie with statement)? Just to be sure and that everybody is aware of the facts: + views don't speed up SQL + both in views and and CTEs, filtering by calculated values may slow down the query unpredictably as predicate pushdown may or may not work as you'd expect. Aggregating should always be done in SQL not by looping over the data as it is transaction save and magnitudes faster. NEVER use invoker rights with a db object (unless you really know what you are doing) If somebody wants to explain to me what the problem is, I am more than willing to listen. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #34 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- The title should be adjusted here, the patches largely make sense, tested combined with 42217 and Claude reported an issue that I confirm Using the test file from 43944, there are multiple entries for a title that differ only in 'Access_Method' - we don't store this, and so there is a change in behavior. Previously, we would add a row for each entry in the file - now we will rewrite each row, so only the last entry for access method 'wins' now and the others are lost so the counts for a single file a drastically reduced -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|ERM Usage – replace |ERM Usage – Database Table |Database Table |erm_usage_yus |erm_usage_yus with a View | Text to go in the|The yearly numbers were |The yearly numbers were release notes|wrong and could not be |wrong and could not be |repaired. |repaired. |The patches replace the | |erm_usage_yus table with a | |view that calculates the | |yearly totals by summing up | |the data from | |erm_usage_mus. | |Dropping the table does not | |result in any data loss. | |The erm_usage_mus View | |remains available for | |reporting, both in the ERM | |module and in the Reports | |module. | --- Comment #35 from Michaela Sieber <michaela.sieber@kit.edu> --- In the COUNTER standard, there are mandatory fields and optional fields that must be explicitly requested. The Access Method is entirely optional and must be explicitly requested. Since this is currently not implemented in Koha at all, as there is no way to specify the required URL parameter, this does not pose a problem for COUNTER API harvesting. It would only become an issue for manually uploaded files containing this data. We also asked in the Mattermost ERM channel whether there are any Koha users who make use of the manual file upload feature. We received only one response, indicating that this is not considered an important feature. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #36 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210612 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210612&action=edit Bug 42218: Import the Regular usage only A COUNTER master report (PR, DR, TR and IR) can hold the usage of both the Regular and the TDM (text and data mining) access method. Koha does not store the access method, so the rows of one title that differ only in Access_Method have the same identity. The second row overwrites the monthly usage of the first, and the yearly usage follows. A SUSHI request for a master report does not filter the access method, and the filter defaults to all, so a harvest can also add TDM usage to the totals. The standard views are preset to Regular and are not affected. Koha now stores the Regular usage only: - the SUSHI request for a master report adds access_method=Regular - the SUSHI response is cleaned of other access methods, for a provider that ignores the filter. In release 5 the items with another Access_Method are dropped. In release 5.1 the Attribute_Performance entries with another Access_Method are dropped, and so is an item that has no entry left - a row of a COUNTER file whose Access_Method is not Regular is not imported: it adds no usage object and no monthly usage. A missing or empty Access_Method is Regular, in any case - one warning on the job page gives the number of rows that were not imported - Koha::ERM::EUsage::SushiCounter->is_regular_access_method holds the rule for both paths The totals that an earlier harvest of a master report stored can include TDM usage. They are not changed: a new harvest of the month replaces them. Test plan: 1. Create a usage data provider and upload a Title Master Report (TR) CSV that has an Access_Method column and these rows for one month: - "Title A", Regular, 5 - "Title A", TDM, 50 - "Title B", TDM, 30 - "Title C", empty Access_Method, 7 2. Open the job of the upload: it has a warning "2 rows with an Access_Method other than Regular were not imported" 3. Open the usage data provider, Titles tab: Titles A and C are there, Title B is not 4. Check the usage of the month and the year: Title A has 5 and not 55, Title C has 7 5. Upload a file with TDM rows only: the job has a warning with the number of rows, and no title and no usage is added 6. Upload a Title Report standard view (for example TR_J1): it imports as before, with no warning 7. Harvest a TR report from a SUSHI server and look at the request in the server log: it has access_method=Regular. A TR_J1 request has no access_method 8. Harvest a TR report from a server that ignores the filter and returns TDM usage: only the Regular usage is stored 9. prove -l t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t Assisted-by: Sonnet 5.5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #37 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210613 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210613&action=edit Bug 42218: Unit tests for the Regular access method CounterFile.t, for release 5 and 5.1, imports a Title Master Report with an Access_Method column. It checks that a title with Regular and TDM rows keeps the Regular count only, in the monthly and in the yearly usage, that a title with TDM rows only is not added, that an empty value and "regular" are imported, and that one warning gives the number of rows that were not imported. A file with TDM rows only adds no usage. SushiCounter.t, for release 5 and 5.1, checks that TDM items and Attribute_Performance entries do not change the COUNTER report body, that a Regular entry in any case is kept, and that a response with TDM usage only gives no rows. COUNTER_5/SushiCounter.t also tests is_regular_access_method on its own. UsageDataProvider.t checks that the URL of a master report has access_method=Regular and that the URL of a standard view has not. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t Assisted-by: Sonnet 5.5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #38 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Access_method differentiates between Regular and TDM (Text and data mining). It is a property that only gets returned from SUSHI if requested in the URL which Koha currently doesn't do. It could only be harvested manually. The patches attached now filter any TDM lines out of a manual report and ensure when requesting from SUSHI that only Regular lines are received. This gives the true usage values - if TDM values are required in future this should be addressed in a follow-up -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210612|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #39 from Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk> --- Created attachment 210670 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210670&action=edit Bug 42218: Import the Regular usage only A COUNTER master report (PR, DR, TR and IR) can hold the usage of both the Regular and the TDM (text and data mining) access method. Koha does not store the access method, so the rows of one title that differ only in Access_Method have the same identity. The second row overwrites the monthly usage of the first, and the yearly usage follows. A SUSHI request for a master report does not filter the access method, and the filter defaults to all, so a harvest can also add TDM usage to the totals. The standard views are preset to Regular and are not affected. Koha now stores the Regular usage only: - the SUSHI request for a master report adds access_method=Regular - the SUSHI response is cleaned of other access methods, for a provider that ignores the filter. In release 5 the items with another Access_Method are dropped. In release 5.1 the Attribute_Performance entries with another Access_Method are dropped, and so is an item that has no entry left - a row of a COUNTER file whose Access_Method is not Regular is not imported: it adds no usage object and no monthly usage. A missing or empty Access_Method is Regular, in any case - one warning on the job page gives the number of rows that were not imported - Koha::ERM::EUsage::SushiCounter->is_regular_access_method holds the rule for both paths The totals that an earlier harvest of a master report stored can include TDM usage. They are not changed: a new harvest of the month replaces them. Test plan: 1. Create a usage data provider and upload a Title Master Report (TR) CSV that has an Access_Method column and these rows for one month: - "Title A", Regular, 5 - "Title A", TDM, 50 - "Title B", TDM, 30 - "Title C", empty Access_Method, 7 2. Open the job of the upload: it has a warning "2 rows with an Access_Method other than Regular were not imported" 3. Open the usage data provider, Titles tab: Titles A and C are there, Title B is not 4. Check the usage of the month and the year: Title A has 5 and not 55, Title C has 7 5. Upload a file with TDM rows only: the job has a warning with the number of rows, and no title and no usage is added 6. Upload a Title Report standard view (for example TR_J1): it imports as before, with no warning 7. prove -l t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t Assisted-by: Sonnet 5.5 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210311|0 |1 is obsolete| | Attachment #210312|0 |1 is obsolete| | Attachment #210313|0 |1 is obsolete| | Attachment #210613|0 |1 is obsolete| | Attachment #210670|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #40 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210708 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210708&action=edit Bug 42218: Create a shared UsageObject class The four usage classes (UsageTitle, UsagePlatform, UsageDatabase and UsageItem) had byte-identical monthly_usages, yearly_usages and erm_usage_*uses methods. This moves them to one base class, Koha::ERM::EUsage::UsageObject, so the yearly usage change in the next patches is made once. Each subclass keeps only _type. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #41 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210709 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210709&action=edit Bug 42218: Rebuild erm_usage_yus from the monthly usage Yearly usage rows were written by CounterFile per COUNTER file: the months of one file were summed and the row was deduplicated on a hash that included the total. A re-harvest of the same year with a different total added a second row, and a year spread over two files gave two partial totals. The yearly numbers were wrong and could not be repaired. erm_usage_yus stays a table, but its rows are now always the sum of erm_usage_mus per year, so erm_usage_mus is the only source of the yearly numbers: - YearlyUsages->rebuild deletes and inserts the rows of one provider, report type and year from the sum of erm_usage_mus over every key column (usage object, provider, year, metric type, access type, yop and report type). It locks the provider row, as the jobs of the months of one provider run in parallel - CounterFile no longer builds or stores yearly usage entries per file. When all rows are done it calls rebuild for each year of the file - UsageObject->yearly_usages is a plain getter, the same as erm_usage_yuses - Koha::ERM::EUsage::YearlyUsage->store and ->delete throw, the object is read-only - the SUSHI harvester job report drops the added_yus and skipped_yus counters and the template no longer shows them Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #42 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210710 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210710&action=edit Bug 42218: Update unit tests The harvester and CounterFile tests no longer count added or skipped yearly rows. They check instead that erm_usage_yus holds the summed totals after an import, and that a second harvest of the same file leaves the yearly count unchanged. YearlyUsage.t covers rebuild: the sum per year and per metric type, a changed or removed month, rows with NULL key columns, other years and report types, the UsageObject getter, to_api, and the read-only store and delete. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/YearlyUsage.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #43 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210711 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210711&action=edit Bug 42218: Unit tests for the Regular access method CounterFile.t, for release 5 and 5.1, imports a Title Master Report with an Access_Method column. It checks that a title with Regular and TDM rows keeps the Regular count only, in the monthly and in the yearly usage, that a title with TDM rows only is not added, that an empty value and "regular" are imported, and that one warning gives the number of rows that were not imported. A file with TDM rows only adds no usage. SushiCounter.t, for release 5 and 5.1, checks that TDM items and Attribute_Performance entries do not change the COUNTER report body, that a Regular entry in any case is kept, and that a response with TDM usage only gives no rows. COUNTER_5/SushiCounter.t also tests is_regular_access_method on its own. UsageDataProvider.t checks that the URL of a master report has access_method=Regular and that the URL of a standard view has not. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t Assisted-by: Sonnet 5.5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #44 from Michaela Sieber <michaela.sieber@kit.edu> --- Created attachment 210712 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210712&action=edit Bug 42218: Import the Regular usage only A COUNTER master report (PR, DR, TR and IR) can hold the usage of both the Regular and the TDM (text and data mining) access method. Koha does not store the access method, so the rows of one title that differ only in Access_Method have the same identity. The second row overwrites the monthly usage of the first, and the yearly usage follows. A SUSHI request for a master report does not filter the access method, and the filter defaults to all, so a harvest can also add TDM usage to the totals. The standard views are preset to Regular and are not affected. Koha now stores the Regular usage only: - the SUSHI request for a master report adds access_method=Regular - the SUSHI response is cleaned of other access methods, for a provider that ignores the filter. In release 5 the items with another Access_Method are dropped. In release 5.1 the Attribute_Performance entries with another Access_Method are dropped, and so is an item that has no entry left - a row of a COUNTER file whose Access_Method is not Regular is not imported: it adds no usage object and no monthly usage. A missing or empty Access_Method is Regular, in any case - one warning on the job page gives the number of rows that were not imported - Koha::ERM::EUsage::SushiCounter->is_regular_access_method holds the rule for both paths The totals that an earlier harvest of a master report stored can include TDM usage. They are not changed: a new harvest of the month replaces them. Test plan: 1. Create a usage data provider and upload a Title Master Report (TR) CSV that has an Access_Method column and these rows for one month: - "Title A", Regular, 5 - "Title A", TDM, 50 - "Title B", TDM, 30 - "Title C", empty Access_Method, 7 2. Open the job of the upload: it has a warning "2 rows with an Access_Method other than Regular were not imported" 3. Open the usage data provider, Titles tab: Titles A and C are there, Title B is not 4. Check the usage of the month and the year: Title A has 5 and not 55, Title C has 7 5. Upload a file with TDM rows only: the job has a warning with the number of rows, and no title and no usage is added 6. Upload a Title Report standard view (for example TR_J1): it imports as before, with no warning 7. prove -l t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t Assisted-by: Sonnet 5.5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off --- Comment #45 from Michaela Sieber <michaela.sieber@kit.edu> --- Signed off using a sandbox -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Lisa G <lgonzalez@palni.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lgonzalez@palni.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #210708|0 |1 is obsolete| | Attachment #210709|0 |1 is obsolete| | Attachment #210710|0 |1 is obsolete| | Attachment #210711|0 |1 is obsolete| | Attachment #210712|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #46 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 211136 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211136&action=edit Bug 42218: Create a shared UsageObject class The four usage classes (UsageTitle, UsagePlatform, UsageDatabase and UsageItem) had byte-identical monthly_usages, yearly_usages and erm_usage_*uses methods. This moves them to one base class, Koha::ERM::EUsage::UsageObject, so the yearly usage change in the next patches is made once. Each subclass keeps only _type. Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #47 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 211137 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211137&action=edit Bug 42218: Rebuild erm_usage_yus from the monthly usage Yearly usage rows were written by CounterFile per COUNTER file: the months of one file were summed and the row was deduplicated on a hash that included the total. A re-harvest of the same year with a different total added a second row, and a year spread over two files gave two partial totals. The yearly numbers were wrong and could not be repaired. erm_usage_yus stays a table, but its rows are now always the sum of erm_usage_mus per year, so erm_usage_mus is the only source of the yearly numbers: - YearlyUsages->rebuild deletes and inserts the rows of one provider, report type and year from the sum of erm_usage_mus over every key column (usage object, provider, year, metric type, access type, yop and report type). It locks the provider row, as the jobs of the months of one provider run in parallel - CounterFile no longer builds or stores yearly usage entries per file. When all rows are done it calls rebuild for each year of the file - UsageObject->yearly_usages is a plain getter, the same as erm_usage_yuses - Koha::ERM::EUsage::YearlyUsage->store and ->delete throw, the object is read-only - the SUSHI harvester job report drops the added_yus and skipped_yus counters and the template no longer shows them Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #48 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 211138 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211138&action=edit Bug 42218: Update unit tests The harvester and CounterFile tests no longer count added or skipped yearly rows. They check instead that erm_usage_yus holds the summed totals after an import, and that a second harvest of the same file leaves the yearly count unchanged. YearlyUsage.t covers rebuild: the sum per year and per metric type, a changed or removed month, rows with NULL key columns, other years and report types, the UsageObject getter, to_api, and the read-only store and delete. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/YearlyUsage.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t yarn cypress run --spec t/cypress/integration/ERM/ModuleDashboard_spec.ts Assisted-by: Sonnet 5 (Anthropic) Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #49 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 211139 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211139&action=edit Bug 42218: Unit tests for the Regular access method CounterFile.t, for release 5 and 5.1, imports a Title Master Report with an Access_Method column. It checks that a title with Regular and TDM rows keeps the Regular count only, in the monthly and in the yearly usage, that a title with TDM rows only is not added, that an empty value and "regular" are imported, and that one warning gives the number of rows that were not imported. A file with TDM rows only adds no usage. SushiCounter.t, for release 5 and 5.1, checks that TDM items and Attribute_Performance entries do not change the COUNTER report body, that a Regular entry in any case is kept, and that a response with TDM usage only gives no rows. COUNTER_5/SushiCounter.t also tests is_regular_access_method on its own. UsageDataProvider.t checks that the URL of a master report has access_method=Regular and that the URL of a standard view has not. Test plan: prove -l t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t Assisted-by: Sonnet 5.5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #50 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 211140 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211140&action=edit Bug 42218: Import the Regular usage only A COUNTER master report (PR, DR, TR and IR) can hold the usage of both the Regular and the TDM (text and data mining) access method. Koha does not store the access method, so the rows of one title that differ only in Access_Method have the same identity. The second row overwrites the monthly usage of the first, and the yearly usage follows. A SUSHI request for a master report does not filter the access method, and the filter defaults to all, so a harvest can also add TDM usage to the totals. The standard views are preset to Regular and are not affected. Koha now stores the Regular usage only: - the SUSHI request for a master report adds access_method=Regular - the SUSHI response is cleaned of other access methods, for a provider that ignores the filter. In release 5 the items with another Access_Method are dropped. In release 5.1 the Attribute_Performance entries with another Access_Method are dropped, and so is an item that has no entry left - a row of a COUNTER file whose Access_Method is not Regular is not imported: it adds no usage object and no monthly usage. A missing or empty Access_Method is Regular, in any case - one warning on the job page gives the number of rows that were not imported - Koha::ERM::EUsage::SushiCounter->is_regular_access_method holds the rule for both paths The totals that an earlier harvest of a master report stored can include TDM usage. They are not changed: a new harvest of the month replaces them. Test plan: 1. Create a usage data provider and upload a Title Master Report (TR) CSV that has an Access_Method column and these rows for one month: - "Title A", Regular, 5 - "Title A", TDM, 50 - "Title B", TDM, 30 - "Title C", empty Access_Method, 7 2. Open the job of the upload: it has a warning "2 rows with an Access_Method other than Regular were not imported" 3. Open the usage data provider, Titles tab: Titles A and C are there, Title B is not 4. Check the usage of the month and the year: Title A has 5 and not 55, Title C has 7 5. Upload a file with TDM rows only: the job has a warning with the number of rows, and no title and no usage is added 6. Upload a Title Report standard view (for example TR_J1): it imports as before, with no warning 7. prove -l t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/SushiCounter.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t Assisted-by: Sonnet 5.5 (Anthropic) Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #51 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Been testing this manually ad with AI this week - there are still issues in the code here, 42217 can skip/overwrite data based on columns we don't fully support. In testing 41294 which I have been asked about by librarians AI found that yearly counts could drift and I tested using the test provider here: https://www.countermetrics.org/celus-test-server/ Just three months of data pulled in had drift as some yearly entries were skipped. I think there's work to continue here, but this and 41294 seem necessary, kicking to RM as I am not ERM expert but have been talking with Michaela about these -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |41294 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41294 [Bug 41294] Allow monthly harvesting only so Data not available for certain months do not result in a complete failure of the harvest -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #52 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 211231 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211231&action=edit Bug 42218: (RM follow-up) Rebuild existing yearly usage Yearly usage stored before the upgrade keeps its old totals until it is imported again. Rebuild erm_usage_yus from erm_usage_mus once, with the same sum as YearlyUsages->rebuild. Test plan: 1) Before applying the patch, upload a COUNTER file for a usage data provider (e.g. attachment 210115 on bug 42217) 2) UPDATE erm_usage_yus SET totalcount = totalcount + 100; 3) SELECT (SELECT SUM(totalcount) FROM erm_usage_yus), (SELECT SUM(usage_count) FROM erm_usage_mus); Confirm the two numbers differ 4) Apply the patch and run updatedatabase 5) Repeat step 3 and confirm the two numbers are equal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #53 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Hi all, I've attached an RM follow-up adding an atomicupdate that rebuilds erm_usage_yus from erm_usage_mus. Without it, the yearly totals stored before the upgrade stay wrong until that provider, report type and year is imported again. Matt is away, and I'd like to push this soon. Nick, could you please sign off the follow-up when convenient? The test plan is in the commit message. About comment 51: as you noted in bug 41294 comment 30, the yearly drift comes from the old per-file yearly entries, which this series replaces with a rebuild from erm_usage_mus. The remaining monthly issues (duplicate rows left by earlier re-harvests, YOP overwrite, missing access_type) are tracked in bug 42217 and bug 43967. I suggest updating this bug's title to: ERM yearly usage totals can be wrong after re-harvesting And also the release notes to: ERM yearly usage totals are now always calculated from the monthly usage, fixing wrong totals when a year was split over several COUNTER files or re-harvested with corrected figures. Existing totals are recalculated during the upgrade. Only regular usage is now imported from COUNTER master reports; text and data mining (TDM) usage is no longer counted. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|The yearly numbers were |The yearly numbers were release notes|wrong and could not be |wrong and could not be |repaired. |repaired. | |ERM yearly usage totals are | |now always calculated from | |the monthly usage, fixing | |wrong totals when a year | |was split over several | |COUNTER files or | |re-harvested with corrected | |figures. Existing totals | |are recalculated during the | |upgrade. Only regular usage | |is now imported from | |COUNTER master reports; | |text and data mining (TDM) | |usage is no longer counted. | |This restriction is | |temporary. Once | |Access_Method is added to | |the database (see Bug | |44050), the restriction to | |Regular can be lifted. Summary|ERM Usage – Database Table |ERM yearly usage totals can |erm_usage_yus |be wrong after | |re-harvesting --- Comment #54 from Michaela Sieber <michaela.sieber@kit.edu> --- Thank you very much, Pedro. We have changed the title and release notes as you suggested, and we have also written a corresponding bug for the missing access_method, Bug 44050, with a detailed justification for restricting the import to regular usage. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=44050 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #211231|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #55 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 211481 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211481&action=edit Bug 42218: (RM follow-up) Rebuild existing yearly usage Yearly usage stored before the upgrade keeps its old totals until it is imported again. Rebuild erm_usage_yus from erm_usage_mus once, with the same sum as YearlyUsages->rebuild. Test plan: 1) Before applying the patch, upload a COUNTER file for a usage data provider (e.g. attachment 210115 on bug 42217) 2) UPDATE erm_usage_yus SET totalcount = totalcount + 100; 3) SELECT (SELECT SUM(totalcount) FROM erm_usage_yus), (SELECT SUM(usage_count) FROM erm_usage_mus); Confirm the two numbers differ 4) Apply the patch and run updatedatabase 5) Repeat step 3 and confirm the two numbers are equal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |26.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42218 --- Comment #56 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Thanks everyone! Pushed to main for 26.11! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org