[Bug 30627] New: koha-run-backups may delete the backup files after work done
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Bug ID: 30627 Summary: koha-run-backups may delete the backup files after work done Change sponsored?: --- Product: Koha Version: 20.11 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: mengu@devinim.com.tr QA Contact: testopia@bugs.koha-community.org CC: robin@catalyst.net.nz I've a doubt about the koha-run-backups script that it deletes its backup files if the backup destination directory is empty. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Mengu Yazicioglu <mengu@devinim.com.tr> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|20.11 |21.11 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Mengu Yazicioglu <mengu@devinim.com.tr> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|koha-run-backups may delete |koha-run-backups delete the |the backup files after work |backup files after finished |done |its job without caring days | |option -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 --- Comment #1 from Mengu Yazicioglu <mengu@devinim.com.tr> --- First findings: and a correction, if there are some other files in the backup destination directory, script also counts them and this makes the problem. In the remove part of script, sed "s:^$name ... part removes the instance name and expects only related backup files, but if somehow there are some other files, backups are being deleted It could be better to find older files with find command maybe. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=23943 CC| |magnus@libriotech.no -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 jabiertxof <jabier.arraiza@marker.es> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jabier.arraiza@marker.es --- Comment #2 from jabiertxof <jabier.arraiza@marker.es> --- I also have this issue, backups are fixed to 2 days even harrcoding days variable from 2 to 7 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Andreas Jonsson <andreas.jonsson@kreablo.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andreas.jonsson@kreablo.se --- Comment #3 from Andreas Jonsson <andreas.jonsson@kreablo.se> --- I do not understand why it is validated that a complete pair of sql.gz and tar.gz exists before deleting the pair. As if it would be especially desirable to preserve backups that may have failed before being completed indefinitely. I suggest that we don't do this. We can select files for deletion with the `find' command. We can use the modification time instead of the date in the filename. These should normally be the same. We can also use a stricter pattern for matching files to avoid deleting files that have not been created by the backup script. For instance: find $backupdir -maxdepth 1 \( -mtime +$days -or -mtime $days \) \ -name $name-'[1-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].???'.gz \ \( -name \*.sql.gz -or -name \*.tar.gz \) -print0 | xargs -0 -r rm -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 --- Comment #4 from Andreas Jonsson <andreas.jonsson@kreablo.se> --- Created attachment 158533 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=158533&action=edit Bug 30627: Verify --days parameter and use find command to select old backups for deletion Test plan * Create some old fake backups: backuproot=/var/spool/koha instance=kohadev backupdir="$backuproot"/"$instance" for i in 1 2 3 4 ; do for j in sql tar xxx ; do file="$backupdir"/"$instance"-$(date -I -d "- $i day").${j}.gz if ! test -e "$file" ; then touch -t "$(date +%Y%m%d%H%M -d "- $i day")" "$file" fi done done * Verify that --days parameter is validated sudo koha-run-backups --days 0 sudo koha-run-backups --days foo * Run backup sudo koha-run-backups --days 3 * Verify that backups from 3 days have been preserved and older backups have been deleted * Verify that filenames that do not match the pattern (the .xxx.gz files) are preserved -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Andreas Jonsson <andreas.jonsson@kreablo.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Assignee|koha-bugs@lists.koha-commun |andreas.jonsson@kreablo.se |ity.org | --- Comment #5 from David Nind <david@davidnind.com> --- Added assignee. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #158533|0 |1 is obsolete| | --- Comment #6 from Chris Cormack <chris@bigballofwax.co.nz> --- Created attachment 159931 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159931&action=edit Bug 30627: Verify --days parameter and use find command to select old backups for deletion Test plan * Create some old fake backups: backuproot=/var/spool/koha instance=kohadev backupdir="$backuproot"/"$instance" for i in 1 2 3 4 ; do for j in sql tar xxx ; do file="$backupdir"/"$instance"-$(date -I -d "- $i day").${j}.gz if ! test -e "$file" ; then touch -t "$(date +%Y%m%d%H%M -d "- $i day")" "$file" fi done done * Verify that --days parameter is validated sudo koha-run-backups --days 0 sudo koha-run-backups --days foo * Run backup sudo koha-run-backups --days 3 * Verify that backups from 3 days have been preserved and older backups have been deleted * Verify that filenames that do not match the pattern (the .xxx.gz files) are preserved Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |victor@tuxayo.net Version|21.11 |master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 --- Comment #7 from Victor Grousset/tuxayo <victor@tuxayo.net> --- I don't understand the code enough to fully QA this but here are some findings. why is there `\( -name \*.sql.gz -or -name \*.tar.gz \)` for the pattern? Doesn't that negates the use of the preceding more strict pattern? Wait, is that just to constrain the ??? to be either sql or tar? (just found that there is an implicit AND (-a) when there is no operator between tests) ----- About switching to using file metadata instead of file name to get the date. There should be at least a mention in the documentation of --days and a mention in the release notes ("Text to go in the release notes"). To lessen the chances of mistakes due to eventual divergences there. So if there is one, we have less to worry about and keep the proposed approach to use metadata. Speaking of divergences, is file metadata something we can count on? I don't know enough about managing servers and backups to reliably say if that's reliable or not. Here are some thoughts - if another backup system backups koha backup files, then we can expect date to be preserved - if the files are moved via ssh or something else to a new machine to restore a koha backup (hardware upgrade for a koha server), then at worse date should be reset to current date, so backups would be deleted later, that doesn't look like much of an issue - zip archives preserve the date - copying on a FAT32 USB drive also - if the place where the koha backup files are mounted is via some network protocol, is that possible (and not that rare enough) that it doesn't support date metadata? I recall with a phone using MTP (not a network protocol and MTP has tons of very bad implementation) that all the files I copied to my PC got their date set to 1970... So that's the only real worry. Does date being reset to 1970 is a thing that can happen enough to compromise the use of date metadata vs name? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl --- Comment #8 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Victor Grousset/tuxayo from comment #7)
So that's the only real worry. Does date being reset to 1970 is a thing that can happen enough to compromise the use of date metadata vs name?
LOL It happens all the time :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #159931|0 |1 is obsolete| | --- Comment #9 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 161494 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=161494&action=edit Bug 30627: Verify --days parameter and use find command to select old backups for deletion Test plan * Create some old fake backups: backuproot=/var/spool/koha instance=kohadev backupdir="$backuproot"/"$instance" for i in 1 2 3 4 ; do for j in sql tar xxx ; do file="$backupdir"/"$instance"-$(date -I -d "- $i day").${j}.gz if ! test -e "$file" ; then touch -t "$(date +%Y%m%d%H%M -d "- $i day")" "$file" fi done done * Verify that --days parameter is validated sudo koha-run-backups --days 0 sudo koha-run-backups --days foo * Run backup sudo koha-run-backups --days 3 * Verify that backups from 3 days have been preserved and older backups have been deleted * Verify that filenames that do not match the pattern (the .xxx.gz files) are preserved Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 --- Comment #10 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Great simplification and fix, Andreas. Victor: Can understand your concerns, but should not really be an issue imo in daily operation. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |24.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|24.05.00 |24.05.00,23.11.03 released in| | CC| |fridolin.somers@biblibre.co | |m Status|Pushed to master |Pushed to stable --- Comment #12 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 23.11.x for 23.11.03 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 --- Comment #13 from Victor Grousset/tuxayo <victor@tuxayo.net> --- (In reply to Marcel de Rooy from comment #10)
Victor: Can understand your concerns, but should not really be an issue imo in daily operation.
Oh great :D I though we were doomed when you said "LOL It happens all the time :)" T_T But no! Thanks for your insight that this shouldn't be a problem in practice ^^ -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Version(s)|24.05.00,23.11.03 |24.05.00,23.11.03,23.05.09 released in| | Status|Pushed to stable |Pushed to oldstable --- Comment #14 from Lucas Gass <lucas@bywatersolutions.com> --- Backported to 23.05.x for upcoming 23.05.09 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Alex Buckley <alexbuckley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |36425 CC| |alexbuckley@catalyst.net.nz Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36425 [Bug 36425] koha-run-backups deletes old backup files based on number of hours since modification - not date of modification -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |36435 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36435 [Bug 36435] Prevent warnings from interrupting koha-run-backups when deleting old backup files -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30627 --- Comment #15 from Aleisha Amohia <aleisha@catalyst.net.nz> --- *** Bug 23943 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org