[Bug 29910] New: Purge old Hea data periodically
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 Bug ID: 29910 Summary: Purge old Hea data periodically Change sponsored?: --- Product: Project Infrastructure Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Hea Assignee: chris@bigballofwax.co.nz Reporter: victor@tuxayo.net QA Contact: jonathan.druart+koha@gmail.com AFAIK it's not purged so we might have misleading information from old libraries than no longer use Koha. So it lessens the trust and the insight Hea gives us. There could be a cron than runs some SQL that deletes submissions that haven't been updated since more than X months. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #1 from Fridolin Somers <fridolin.somers@biblibre.com> --- Sure. This is why we see versions like 3.14, 3.22 ^^ -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #2 from Victor Grousset/tuxayo <victor@tuxayo.net> --- That wouldn't be a surprise to have such old production instances never updated. However 3.18.x is the oldest branch with Hea available so seeing 3.14 and 3.09 (?!) in the data makes no sense XD I see that Hea is hosted by BibLibre, good thing you stumbled on this ticket Fridolin ^^ So, how feasible would it be to have a cron job that runs an SQL delete query? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|chris@bigballofwax.co.nz |jonathan.druart+koha@gmail. | |com Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #3 from Jonathan Druart <jonathan.druart+koha@gmail.com> ---
select year(modification_time) as yyyy, count(*) from installation group by yyyy; +------+----------+ | yyyy | count(*) | +------+----------+ | 2014 | 8 | | 2019 | 114 | | 2020 | 2726 | | 2021 | 5893 | | 2022 | 4987 | +------+----------+
What do we remove? What does "periodically" mean? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I think it's supposed to report every month... so maybe we could keep a timeframe of like 6 months back? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #5 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Katrin Fischer from comment #4)
I think it's supposed to report every month... so maybe we could keep a timeframe of like 6 months back?
Assuming the date updates if a new report comes in - or is it only updated if data actually changes (like a timestamp on the database)? We'd want the date last reported I think. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #6 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #3)
select year(modification_time) as yyyy, count(*) from installation group by yyyy; +------+----------+ | yyyy | count(*) | +------+----------+ | 2014 | 8 | | 2019 | 114 | | 2020 | 2726 | | 2021 | 5893 | | 2022 | 4987 | +------+----------+
What do we remove? What does "periodically" mean?
Nothing between 2014 and 2019 is a little bit odd? I agree with Katrin: 6 months or perhaps 12 months to be safe should be the cutoff. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #7 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Marcel de Rooy from comment #6)
(In reply to Jonathan Druart from comment #3)
select year(modification_time) as yyyy, count(*) from installation group by yyyy; +------+----------+ | yyyy | count(*) | +------+----------+ | 2014 | 8 | | 2019 | 114 | | 2020 | 2726 | | 2021 | 5893 | | 2022 | 4987 | +------+----------+
What do we remove? What does "periodically" mean?
Nothing between 2014 and 2019 is a little bit odd?
I've purged them already some years ago. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #8 from Fridolin Somers <fridolin.somers@biblibre.com> --- In my opinion keeping 1 year is a good compromise. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #9 from David Cook <dcook@prosentient.com.au> --- Another thought... rather than purging the data... make it more obvious on the Hea webapp what numbers belong to what year? That could be really interesting as you could see changes in data over time. Then it might be worthwhile keeping maybe the last 5 years of data, if long-term storage of data is a problem. Of course, numbers can also be misleading. A drop in numbers in more recent years might make it seem like fewer people using Koha when really it's just fewer people reporting their usage... But anyway, that's my 2 cents. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #10 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to David Cook from comment #9)
Another thought... rather than purging the data... make it more obvious on the Hea webapp what numbers belong to what year?
How? Who? When? I am not willing to work on Hea improvements. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Besides numbers we created Hea to see what people are using etc. If a Koha installation stops sending, we should probably assume it's no longer active or wanting to share data. I think from this perspective we should definitely purge data. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #12 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Katrin Fischer from comment #11)
Besides numbers we created Hea to see what people are using etc. If a Koha installation stops sending, we should probably assume it's no longer active or wanting to share data. I think from this perspective we should definitely purge data.
Agree completely. Lets not add new features here but have recent data. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #13 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I am also not interested in new features, and this is for a separate bug, but I keep thinking if it's ok to give them no option to delete their data, especially library name/address information. Or maybe it's possible and I am just not aware? So a year is already quite long, but I am ok with it, even tho I'd prefer something shorter. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #10)
(In reply to David Cook from comment #9)
Another thought... rather than purging the data... make it more obvious on the Hea webapp what numbers belong to what year?
How? Who? When?
I am not willing to work on Hea improvements.
Fair enough. I'm not willing to work on it either. I always think of it as a BibLibre project, so I figure they might be willing to make some improvements. Actually, I see a little problem with the website. Maybe I'll try to contribute a little change... hehe. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marion.durand@biblibre.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #15 from Victor Grousset/tuxayo <victor@tuxayo.net> ---
thinking if it's ok to give them no option to delete their data, especially library name/address information.
It depends on how Hea works when receiving new data from and existing library. If it overwrites the existing record, then removing library name&address and resubmitting the data Hea allows to get unlisted. (and remove lib name&address from the DB) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution|--- |FIXED --- Comment #16 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Added the following cronjob: 0 1 1 * * /usr/bin/mysqldump heastats | gzip > /home/hea/dumps/$(date +%Y-%m-%d)-heastats.sql.gz && /usr/bin/mysql -e "DELETE FROM installation WHERE modification_time < DATE_SUB(NOW(), INTERVAL 6 MONTH);" It will dump the DB and remove all installations without changes in the last 6 months. First (manual) run removed 6123 installations. 8744 still in DB. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #17 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Great! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29910 --- Comment #18 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Amazing thanks! Now oldest reported version is 16.11 instead of 3.09! Went from 54 million patrons to 40 million. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org