[Bug 34587] New: Add a Usage Statistics module to ERM
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Bug ID: 34587 Summary: Add a Usage Statistics module to ERM Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: ERM Assignee: jonathan.druart+koha@gmail.com Reporter: matt.blenkinsop@ptfs-europe.com CC: jonathan.druart+koha@gmail.com, jonathan.field@ptfs-europe.com, martin.renvoize@ptfs-europe.com, matt.blenkinsop@ptfs-europe.com, pedro.amorim@ptfs-europe.com A module has been developed to support the gathering and analysis of usage statistics for e-resources in the ERM module. Github link is attached, here is also a video to a demonstration of the module for context on how it works: https://www.youtube.com/live/Ev1tJt7yUjc?si=BwgDATEt4JlM6Yzj&t=5658 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- URL| |https://github.com/PTFS-Eur | |ope/koha/tree/bug_34587 --- Comment #1 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- https://github.com/PTFS-Europe/koha/tree/bug_34587 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|jonathan.druart+koha@gmail. |matt.blenkinsop@ptfs-europe |com |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Large patch Severity|enhancement |new feature Change sponsored?|--- |Sponsored Depends on| |34497, 34468, 34448 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34448 [Bug 34448] ERM should be able to display error messages coming from the API https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34468 [Bug 34468] Add a progress callback to job_progress.js https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34497 [Bug 34497] Vue - Dialog component should allow for confirmation input options -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #2 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- This is ready for testing Testing notes: Credentials will be required to harvest data via SUSHI, please check if your company/organisation has any access credentials that can be used for testing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |brendan@bywatersolutions.co | |m, | |fridolin.somers@biblibre.co | |m, | |katrin.fischer@bsz-bw.de, | |kyle@bywatersolutions.com, | |nick@bywatersolutions.com, | |paul.poulain@biblibre.com, | |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #3 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- Unit tests are still in development and will be following shortly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Jonathan Field <jonathan.field@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jzairo@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #4 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- A sandbox is now live at this link: http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/erm/eusage/... Credentials will be required for testing, please check if your organisation has access to any valid SUSHI credentials in order to test. All credentials are redacted in the sandbox and will not be visible to other users. In the released version, credentials will be visible in the ERM module. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Pedro Amorim <pedro.amorim@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #5 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I feel like should advertise this a bit more to get more testing - a mail to the mailing list maybe? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |emily.lamancusa@montgomeryc | |ountymd.gov -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #6 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (quick note) test_connection returns 500 if credentials are not correct (I guess it's the problem). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Eric Phetteplace <ephetteplace@cca.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ephetteplace@cca.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Amit Gupta <amitddng135@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amit.gupta@informaticsgloba | |l.com, | |amitddng135@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA CC| |lucas@bywatersolutions.com --- Comment #7 from Lucas Gass <lucas@bywatersolutions.com> --- When I enable ERM and Add a new data provider manually I get a type error: this.$refs.table.dt is not a function. For me this comes from KohaTable.vue on line 150: let dt = this.$refs.table.dt() I have tried restarting everything and rebuilding the js with yarn js:build. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #8 from Jessie Zairo <jzairo@bywatersolutions.com> --- Created attachment 156581 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=156581&action=edit queued-report-fail -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #9 from Jessie Zairo <jzairo@bywatersolutions.com> --- We successfully created a new data provider via SUSHI. We got a successful test - Harvester connection was successful for usage data provider. When we run the job for report type TR_J1, we get a message that the report goes to queue, but fails. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #10 from Jonathan Field <jonathan.field@ptfs-europe.com> --- (In reply to Jessie Zairo from comment #9)
We successfully created a new data provider via SUSHI. We got a successful test - Harvester connection was successful for usage data provider. When we run the job for report type TR_J1, we get a message that the report goes to queue, but fails.
Hi Jessie, is this on our sandbox? If so, we found a problem earlier today with the background jobs which we are looking into. I’ll let you know when resolved. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #11 from Jonathan Field <jonathan.field@ptfs-europe.com> --- (In reply to Jessie Zairo from comment #9)
We successfully created a new data provider via SUSHI. We got a successful test - Harvester connection was successful for usage data provider. When we run the job for report type TR_J1, we get a message that the report goes to queue, but fails.
We've fixed our sandbox now, apologies for that, we were missing some of the very latest code which is there now :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Jessie Zairo <jzairo@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off --- Comment #12 from Jessie Zairo <jzairo@bywatersolutions.com> --- Testing Plan 1.) Make sure the ERMModule system preference is set to Enable 2.) Navigate to E-resource management --> eUsage --> Data providers 3.) Click + New data provider 4.) Begin typing the data provider name (vendor) that you wish to set up. If a match is found, select the data provider. If it is not found, select the Create manually button to the right. 5.) Enter any additional information about the data provider in the description. (Not a mandatory field) 6.) Select the Harvester status to Active or Inactive. (Not a mandatory field) 7.) Select the report type from the drop-down menu. 8.) Under SUSHI credentials, you will see the Service URL. If the data provider was created from the registry, this field will be filled in. If you entered the data provider manually you will need to enter the Service URL. 9.) If you are connecting via SUSHI you will need to enter the Report Release (this is the counter type). You can find more information at https://www.projectcounter.org/code-of-practice-five-sections/5-delivery-cou.... 5 is the standard. 10.) Enter the customer id and requestor id for the sushi reports 11.) Click submit. 12.) Under the Actions column click test (if you entered SUSHI credentials). Confirm Harvester connection was successful for the usage data provider 13.) Click Run now. Enter a beginning date and end date. Click Yes, Run. 14.) You will see a Job for report type (of the report) at the top of the page. Make sure the job has been queued, click the 'click here' to check its progress. 15.) Confirm the report has run. 16.) Click on the Name of the data provider in the table, if you manually created the data provider start at this step. You will see 7 tabs across the top (detail, titles, platforms, items, databases, manual upload, import logs. 17.) If you manually created a data provider, select the manual upload tab. Upload a counter 5 report. Confirm the titles load. 18.) On the right-hand side click on Reports under eUsage. 19.) Click on Create Report. 20.) Select the required fields and data needed for the report. Select report columns. Enter the name of your report at the bottom of the page and click save report. Click submit. 21.) Confirm report runs 22.) Click on Saved Reports and confirm your report saved. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #13 from Jessie Zairo <jzairo@bywatersolutions.com> --- Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Stephanie <stephanie.sopka@montgomerycountymd.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |stephanie.sopka@montgomeryc | |ountymd.gov --- Comment #14 from Stephanie <stephanie.sopka@montgomerycountymd.gov> --- Created attachment 157101 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157101&action=edit screenshot of problem -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #15 from Stephanie <stephanie.sopka@montgomerycountymd.gov> --- I entered our library's Ebsco credentials as they are currently set up in Coral and working correctly. When I test the connection here, I get a message saying "No connection for usage data provider EBSCO Information Services, please check your credentials and try again". However, I ran the request anyway, and got the reports I asked for in the Import log. Everything seems to have worked regardless of the message saying there's no connection. I also entered our ProQuest credentials, and didn't have the same issue. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #16 from Stephanie <stephanie.sopka@montgomerycountymd.gov> --- For uploading data, the current Coral manual imports require that the header information from a report be removed before upload. I am not sure how this will interact with the way files have to be imported for the ERM module. I believe the goal is for information from Coral to be transferred into the ERM without too much fussing, so just wanting to note this difference. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #17 from Lucas Gass <lucas@bywatersolutions.com> --- (In reply to Lucas Gass from comment #7)
When I enable ERM and Add a new data provider manually I get a type error:
this.$refs.table.dt is not a function. For me this comes from KohaTable.vue on line 150:
let dt = this.$refs.table.dt()
I have tried restarting everything and rebuilding the js with yarn js:build.
I don't think this was addressed and I am still getting this error. To recreate: 1. ERM -> Add new data provider 2. Create manually 3. Add a provider manually 4. Go back to /cgi-bin/koha/erm/agreements 5. Notice the type error -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #18 from Jessie Zairo <jzairo@bywatersolutions.com> --- Feedback from a library partner "I was able to play around a little, and it is looking good." Data Providers Adding + New data provider Created a new provider by finding an existing one. Service URL was automatically generated. Added our institution’s information. Able to save without problems. Able to edit without problems. Created a new provider by manually inputting information. Manually added service URL Added our institution’s information. Able to save without problems. Able to edit without problems. NOTE: fields are grayed out unless you click “create manually” button to enable them. Connection Test Both registry and manual providers connected to the SUSHI server successfully. Harvest Run Both registry and manual providers harvested most of the requested reports. There were a few reports that didn’t return with data. Reports that returned data were added successfully to the provider. Manual Upload I tried a report that I thought was counter 5 but it didn’t upload. I didn’t get a chance to get another report to test further. Reports Able to create new reports with no errors. Able to run saved reports with no errors. Able to export displayed data from the report. NOTE: Only visible data rows were downloaded. Must select show all to get the full data downloaded. Noticed a discrepancy between the harvested data and the report data. Downloaded the import log data and compared the titles against the exported report data, in my SpringerLink test, I saw a difference of about 115 or so titles. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |michaela.sieber@kit.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #19 from Michaela Sieber <michaela.sieber@kit.edu> --- First of all thank you for the work you already did! I added two of our main providers = Springer and De Gruyter Testing results: -- HARVESTING -- Harvesting of about 190.000 rows for SpringerLink was done with some errors, e.g. Building the Cycling City could not be processed - please check the logs Statistische Datenanalyse mit SPSS für Windows could not be processed - please check the logs Concise Manual of Hematology and Oncology could not be processed - please check the logs See http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/admin/background_jobs.pl?op=view&id=40 It would be helpful if you could provide the number of titles successfully added/that could not be processed in the Job report similar to the number of usages added/skipped. The same with De Gruyter, many titles could not be processed. -- REPORTS -- Feedback on sorting reports tables: E-resource management > eUsageReports > View report I tried to sort by month March but the table was not sorted, just processing.... It would be great to have a 'Reporting_Period_Total' column -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #20 from Pedro Amorim <pedro.amorim@ptfs-europe.com> --- (In reply to Lucas Gass from comment #17)
(In reply to Lucas Gass from comment #7)
When I enable ERM and Add a new data provider manually I get a type error:
this.$refs.table.dt is not a function. For me this comes from KohaTable.vue on line 150:
let dt = this.$refs.table.dt()
I have tried restarting everything and rebuilding the js with yarn js:build.
I don't think this was addressed and I am still getting this error.
To recreate:
1. ERM -> Add new data provider 2. Create manually 3. Add a provider manually 4. Go back to /cgi-bin/koha/erm/agreements 5. Notice the type error
Hi Lucas, thank you! I'm unable to reproduce this. I'm assuming /cgi-bin/koha/erm/agreements is a typo but I can't reproduce the error in either agreements or data providers. I created this manual data provider: http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/erm/eusage/... But I have no error anywhere, are you able to provide additional instructions? Also, update your branch if you're testing locally. We've pushed quite a few bug fixes since the first submission here. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #21 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- (In reply to Michaela Sieber from comment #19) Hi Michaela, Thanks for testing!
Testing results:
-- HARVESTING --
Harvesting of about 190.000 rows for SpringerLink was done with some errors, e.g. Building the Cycling City could not be processed - please check the logs Statistische Datenanalyse mit SPSS für Windows could not be processed - please check the logs Concise Manual of Hematology and Oncology could not be processed - please check the logs
The fact that the rest of the titles are harvesting successfully leads me to suspect that these failures will be due to the fields in the database not being long enough to support the data being harvested. We recently increased the size of a number of fields across the different data types but there are probably more that need doing. We will investigate this and extend fields where appropriate to give some good margin for error when harvesting.
See http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/admin/ background_jobs.pl?op=view&id=40
It would be helpful if you could provide the number of titles successfully added/that could not be processed in the Job report similar to the number of usages added/skipped.
That can certainly be looked at - we'll add this to the list but this will probably follow later after any bug fixes are tackled
-- REPORTS -- Feedback on sorting reports tables: E-resource management > eUsageReports > View report
I tried to sort by month March but the table was not sorted, just processing....
This is a known issue that we're looking at, hopefully we can patch this soon!
It would be great to have a 'Reporting_Period_Total' column
You can choose to do a report "By month with period total" in the custom reports viewer, this will add the column you mention :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #22 from Lucas Gass <lucas@bywatersolutions.com> ---
Hi Lucas, thank you! I'm unable to reproduce this. I'm assuming /cgi-bin/koha/erm/agreements is a typo but I can't reproduce the error in either agreements or data providers. I created this manual data provider: http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/erm/eusage/ usage_data_providers/18 But I have no error anywhere, are you able to provide additional instructions?
Also, update your branch if you're testing locally. We've pushed quite a few bug fixes since the first submission here.
I am testing locally, have updated my branch, ran yarn. However I am not going to hold this up due to something that may very well be an issue I am only experiencing locally. I will reset to 'Signed off'. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Pedro Amorim <pedro.amorim@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35120 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35120 [Bug 35120] ERM - We need tests for Dialog inputs -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #23 from Pedro Amorim <pedro.amorim@ptfs-europe.com> --- (In reply to Matt Blenkinsop from comment #21)
(In reply to Michaela Sieber from comment #19)
Hi Michaela,
Thanks for testing!
Testing results:
-- HARVESTING --
Harvesting of about 190.000 rows for SpringerLink was done with some errors, e.g. Building the Cycling City could not be processed - please check the logs Statistische Datenanalyse mit SPSS für Windows could not be processed - please check the logs Concise Manual of Hematology and Oncology could not be processed - please check the logs
The fact that the rest of the titles are harvesting successfully leads me to suspect that these failures will be due to the fields in the database not being long enough to support the data being harvested. We recently increased the size of a number of fields across the different data types but there are probably more that need doing. We will investigate this and extend fields where appropriate to give some good margin for error when harvesting.
It appears this was indeed the case, fixed by this commit: https://github.com/PTFS-Europe/koha/commit/f0608a0dcaa2d5fa0875603d08124fb10...
See http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/admin/ background_jobs.pl?op=view&id=40
It would be helpful if you could provide the number of titles successfully added/that could not be processed in the Job report similar to the number of usages added/skipped.
That can certainly be looked at - we'll add this to the list but this will probably follow later after any bug fixes are tackled
This is something that I believe was working before, but got broken when I moved everything under EUsage and created a new SushiCounter.pm class. However, I believe this is now fixed, you should see a "Titles added" row now in the job report if titles, "Databases" for databases, etc. Fixed by this commit: https://github.com/PTFS-Europe/koha/commit/42fad0948a1318d254e7133925b3121f2... @Michaela would you kindly test again and verify for yourself? Particularly the 190.000 rows provider if possible. Your sign-off here would be very much appreciated, as others of course. We've provisioned a new sandbox at: http://staff-sushi.sandboxes.ptfs-europe.co.uk/ With the most up to date code from: https://github.com/PTFS-Europe/koha/commits/bug_34587 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #24 from Michaela Sieber <michaela.sieber@kit.edu> --- As far as I can see all titles are harvested now. But the usage per month is not correct in my report. First run 01.-31.01.2023 Second run 01.02.-30.04.2023 Example: Nature (Online ISSN 1476-4687) normally 3000-4000 Unique_Item_Requests per month. Unique_Item_Requests should be: Jan23: 3431 Feb23: 3159 Mar23: 3758 Apr23: 2775 Check Import Logs on Data provider Springer and Download file. The usage numbers for Nature (=ISSN 1476-4687) for Platform nature.com are correct in the downloaded file . Check now the created J1 report See that the usage numbers for February and March are wrong Unique_Item_Requests Feb23: 20 Mar23: 6 usage numbers for Jan and April are correct -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #25 from Michaela Sieber <michaela.sieber@kit.edu> --- (In reply to Michaela Sieber from comment #24) In a Third run everything was correct Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #26 from Pedro Amorim <pedro.amorim@ptfs-europe.com> --- (In reply to Michaela Sieber from comment #24)
As far as I can see all titles are harvested now.
But the usage per month is not correct in my report. First run 01.-31.01.2023 Second run 01.02.-30.04.2023
Example: Nature (Online ISSN 1476-4687)
normally 3000-4000 Unique_Item_Requests per month.
Unique_Item_Requests should be: Jan23: 3431 Feb23: 3159 Mar23: 3758 Apr23: 2775
Check Import Logs on Data provider Springer and Download file. The usage numbers for Nature (=ISSN 1476-4687) for Platform nature.com are correct in the downloaded file .
Check now the created J1 report
See that the usage numbers for February and March are wrong Unique_Item_Requests Feb23: 20 Mar23: 6
usage numbers for Jan and April are correct
Hi Michaela. Once again, thank you so much for this invaluable testing (and sign-off!). We detected and fixed 2 issues here: 1) The problem seems to have been because the 2 titles titled 'nature' in the first report have the same 1476-4687 ISSN and Proprietary_ID, in fact the only thing that differs between the two is the Platform. This was causing the 'nature' title to match erroneously. This is the first time we've seen this and we improved our matching rules: https://github.com/PTFS-Europe/koha/commit/a855ea64e5b97262bc9a89826020fc43a... The problem was even bigger for the period of jan to april 2023. Because here we have 3 instances of 'nature' 1476-4687 ISSN, but all 3 are from different platforms. With the above fix, everything should be correct in the first harvest, as well as all subsequent harvests. These 3 'nature' titles are effectively 3 different titles in Koha and each have their corresponding correct monthly usages. I've reset the sandbox with these fixes if you're available to test it once again. 2) One issue in the logs detected was 'title' column was too short, we changed it from varchar(255) to mediumtext to address this: https://github.com/PTFS-Europe/koha/commit/32f6bbf4bc79d07a9815bbaa17756a9ca... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #27 from Michaela Sieber <michaela.sieber@kit.edu> --- (In reply to Pedro Amorim from comment #26)
Great, Thanks for fixing and for your explanations. Fingers crossed for 23.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Bug 34587 depends on bug 34468, which changed state. Bug 34468 Summary: Add a progress callback to job_progress.js https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34468 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Bug 34587 depends on bug 34497, which changed state. Bug 34497 Summary: Vue - Dialog component should allow for confirmation input options https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34497 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #28 from Nick Clemens <nick@bywatersolutions.com> --- Did a run through the code, a little clean up for QA tools - talked with the PFTS team and RM and our testers, all seems to be working well. Standalone module so no worries about the late push. Passing QA! Excellent work all! My SO lines are added on the remote branch. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_23_11_candidate -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |23.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #29 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35218 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35218 [Bug 35218] No tests for /erm/sushi_service -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35219 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35219 [Bug 35219] ERM usage endpoints not showing up in documentation -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35229 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35229 [Bug 35229] Add further cypress tests for Usage reporting -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Aude Charillon <aude.charillon@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Documentation| |Jonathan Field contact| | CC| |aude.charillon@ptfs-europe. | |com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35258 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35258 [Bug 35258] Updating po files locally fails -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #30 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Why those commented lines? Should we remove? koha-tmpl/intranet-tmpl/prog/js/vue/components/ERM/UsageStatisticsDataProvidersFormAdd.vue 108 <!-- <li> 109 <label for="usage_data_provider_method" 110 >{{ $__("Method") }}: 111 </label> 112 <input 113 id="usage_data_provider_method" 114 v-model="usage_data_provider.method" 115 :disabled="!selected_provider && !manual_form" 116 /> 117 </li> 118 <li> 119 <label for="usage_data_provider_aggregator" 120 >{{ $__("Aggregator") }}: 121 </label> 122 <input 123 id="usage_data_provider_aggregator" 124 v-model="usage_data_provider.aggregator" 125 :disabled="!selected_provider && !manual_form" 126 /> 127 </li> --> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #31 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- koha-tmpl/intranet-tmpl/prog/js/vue/components/ERM/UsageStatisticsDataProvidersFormAdd.vue: fp_config: flatpickr_defaults, This is not needed, as well as the flatpickr import. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #32 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- Created attachment 158754 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=158754&action=edit Bug 34587: (follow-up) Remove unnecessary code This patch removes some commented out lines and some unnecessary imports. No impact to working code - self signoff Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|rel_23_11_candidate | --- Comment #33 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Follow-up pushed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed --- Comment #34 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- ERM/UsageReports_spec.ts and ERM/DataProviders_spec.ts are failing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #35 from Pedro Amorim <pedro.amorim@ptfs-europe.com> --- (In reply to Jonathan Druart from comment #34)
ERM/UsageReports_spec.ts and ERM/DataProviders_spec.ts are failing
See bug 35229. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35374 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35374 [Bug 35374] Translations contain config from ERM/data providers -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35392 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35392 [Bug 35392] HTML in translatable string -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Pedro Amorim <pedro.amorim@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35418 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35418 [Bug 35418] SUSHI harvest hangs -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This feature adds the release notes| |ability to create Data | |Providers and harvest usage | |data in COUNTER format. | |Data Providers (or Data | |Platforms) are the | |organisations who provide | |usage statistics for your | |electronic resources. This | |could be a vendor or a | |platform provider. The data | |provider record contains | |information about SUSHI | |credentials and any COUNTER | |data which has been | |harvested from the | |provider. There is also the | |ability to create custom | |reports using this COUNTER | |data. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35375 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35375 [Bug 35375] Capitalization: Report Release -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #36 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Trailing space chars at t/cypress/support/e2e.js:161 Looks like we need to improve the QA checks. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Needs documenting -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Aude Charillon <aude.charillon@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/751 Resolution|--- |FIXED Status|Needs documenting |RESOLVED --- Comment #37 from Aude Charillon <aude.charillon@ptfs-europe.com> --- Documented by Jonathan Field. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37288 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37288 [Bug 37288] Edit data provider form does not show the name -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl --- Comment #38 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- It seems that this patch set added two inconsistencies in the db schema: STRUCTURE `date_uploaded` timestamp NULL DEFAULT current_timestamp() COMMENT 'counter file upload date', DBREV 230600056.pl `date_uploaded` timestamp DEFAULT NULL DEFAULT current_timestamp() COMMENT 'counter file upload date', => You see two DEFAULTs in the dbrev. Probably a typo. STRUCTURE `importdate` timestamp NULL DEFAULT current_timestamp() COMMENT 'counter file import date', DBREV 230600056. `importdate` timestamp DEFAULT NULL DEFAULT current_timestamp() COMMENT 'counter file import date', => Same Could the authors give follow-up and correct please? Thanks. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #39 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- Created attachment 177672 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=177672&action=edit Bug 34587: (follow-up) Fix DB inconsistencies -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |RM_priority -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #40 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Matt Blenkinsop from comment #39)
Created attachment 177672 [details] [review] Bug 34587: (follow-up) Fix DB inconsistencies
Hi Matt, can I bother you to put this ona new bug please? We'll need it for the release notes and backporting. Thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 --- Comment #41 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- (In reply to Katrin Fischer from comment #40)
(In reply to Matt Blenkinsop from comment #39)
Created attachment 177672 [details] [review] [review] Bug 34587: (follow-up) Fix DB inconsistencies
Hi Matt, can I bother you to put this ona new bug please? We'll need it for the release notes and backporting. Thanks!
Done - https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39075 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|RM_priority | --- Comment #42 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Matt Blenkinsop from comment #41)
(In reply to Katrin Fischer from comment #40)
(In reply to Matt Blenkinsop from comment #39)
Created attachment 177672 [details] [review] [review] [review] Bug 34587: (follow-up) Fix DB inconsistencies
Hi Matt, can I bother you to put this ona new bug please? We'll need it for the release notes and backporting. Thanks!
Done - https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39075
Thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34587 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |39075 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39075 [Bug 39075] Fix DB inconsistencies in the usage statistics module -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org