[Bug 43638] New: Add core support for scheduled/automated patron image import
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 Bug ID: 43638 Summary: Add core support for scheduled/automated patron image import Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Patrons Assignee: koha-bugs@lists.koha-community.org Reporter: martin.renvoize@openfifth.co.uk QA Contact: testopia@bugs.koha-community.org CC: gmcharlt@gmail.com, kyle@bywatersolutions.com Target Milestone: --- Depends on Bug 43637 (refactor patron image upload into a reusable engine class and background job), which this builds on. This bug is the scheduling half of that work: automatically importing patron images from a remote/local file source on a schedule, the same way Bug 43616 added scheduled/automated patron record import on top of the existing, unmodified Koha::Patrons::Import engine. Planned approach: - New patron_image_import_accounts and patron_image_import_log tables, plus Koha::PatronImageImportAccount(s)/Koha::PatronImportLog(s) object classes, mirroring Bug 43616's own account/log model (name, file transport, matchpoint, active flag, per-(account, filename) idempotency). - A new Tools page ("Schedule patron image imports") for configuring accounts - a smaller option surface than the patron-import case, since image import has no equivalent of default values or preserved fields. - A new misc/cronjobs/patron_image_imports.pl cronjob polling each active account's configured Koha::File::Transport, calling Koha::Patron::Image::Import->import_batch (from Bug 43637) for zip archives it finds, and also handling loose image files sitting directly on the transport (matched by filename stem against the account's chosen matchpoint) - not just zips, since an unattended feed may not always package images into an archive. No code has been written yet for this bug - filing it now to track the plan and its dependency on Bug 43637 before design/implementation begins. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |43637 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43637 [Bug 43637] Refactor patron image upload into a reusable engine class and background job -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |martin.renvoize@openfifth.c |ity.org |o.uk -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #1 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206826 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206826&action=edit Bug 43638: Automated Schema Update Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #2 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206827 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206827&action=edit Bug 43638: Add is_boolean annotation to PatronImageImportAccount Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #3 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206828 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206828&action=edit Bug 43638: Add Koha::PatronImageImportAccount(s) and Koha::PatronImageImportLog(s) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #4 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206829 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206829&action=edit Bug 43638: Add MAX_ZIP_MEMBERS cap to Koha::Patron::Image::Import This commit adds a safety cap to prevent extraction and processing of zip archives with more than 1000 non-directory members, bounding worst-case extraction work from oversized or malicious zip files. The cap is enforced in import_batch before any extraction occurs, and returns a TOO_MANY_MEMBERS error when exceeded. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206830 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206830&action=edit Bug 43638: Add patron_image_imports.pl cron job Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #6 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206831 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206831&action=edit Bug 43638: Add 'Schedule patron image imports' page under Tools Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #7 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206832 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206832&action=edit Bug 43638: Register patron_image_imports.pl in the default crontab templates Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37248 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37248 [Bug 37248] [Omnibus] Power to the user -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37248 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37248 [Bug 37248] [Omnibus] Power to the user -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA CC| |andrew@bywatersolutions.com --- Comment #8 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Can you please add a test plan here? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #206826|0 |1 is obsolete| | Attachment #206827|0 |1 is obsolete| | Attachment #206828|0 |1 is obsolete| | Attachment #206829|0 |1 is obsolete| | Attachment #206830|0 |1 is obsolete| | Attachment #206831|0 |1 is obsolete| | Attachment #206832|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #9 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209015 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209015&action=edit Bug 43638: Automated Schema Update Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #10 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209016 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209016&action=edit Bug 43638: Add is_boolean annotation to PatronImageImportAccount Mark the active column of patron_image_import_accounts as a boolean in the DBIC result class, below the auto-generated part, so that it is handled as a true/false value rather than a number. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #11 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209017 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209017&action=edit Bug 43638: Add Koha::PatronImageImportAccount(s) and Koha::PatronImageImportLog(s) Add the Koha::Object(s) classes for the two new tables: - Koha::PatronImageImportAccount(s): an import account, i.e. the file transport to poll for patron images and the field used to match the files to patrons (the matchpoint). file_transport() returns the transport as its concrete backend (local, FTP or SFTP), and resolved_matchpoint() gives the matchpoint without the "patron_attribute_" prefix used for patron attributes. - Koha::PatronImageImportLog(s): one row per account and file name that was processed, with the outcome (processed or error) and its details. To test: 1. Run $ prove t/db_dependent/Koha/PatronImageImportAccounts.t \ t/db_dependent/Koha/PatronImageImportLogs.t => Both pass Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #12 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209018 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209018&action=edit Bug 43638: Add MAX_ZIP_MEMBERS cap to Koha::Patron::Image::Import This commit adds a safety cap to prevent extraction and processing of zip archives with more than 1000 non-directory members, bounding worst-case extraction work from oversized or malicious zip files. The cap is enforced in import_batch before any extraction occurs, and returns a TOO_MANY_MEMBERS error when exceeded. To test: 1. Run $ prove t/db_dependent/Koha/Patron/Image/Import.t => It passes, including the test of the cap on the number of members of a zip archive Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #13 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209019 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209019&action=edit Bug 43638: Add patron_image_imports.pl cron job Add misc/cronjobs/patron_image_imports.pl. It polls the file transport of every active patron image import account for new zip archives and loose image files, and imports them with Koha::Patron::Image::Import (Bug 43637), matching the images to patrons with the matchpoint of the account (card number, username or a unique patron attribute). Each file is moved on the file transport to a "processed" or an "error" directory, and logged once per account and file name in patron_image_import_log, so that running the cron job repeatedly is safe. Accounts with an invalid matchpoint, or that cannot connect, are skipped and the problem is logged. Without -c it only lists what it would do. To test: You need Bug 43637 and all the patches of this bug. Everything below is done in KTD. 1. Apply the patches and, in "ktd --shell", run updatedatabase restart_all 2. Create a file transport that reads from a local directory. The Administration > File transports form only offers FTP and SFTP, so, if you have no FTP or SFTP server to use, create a "local" transport directly. In "ktd --shell": mkdir -p /tmp/patron-images then in "ktd --dbshell" (outside the shell): INSERT INTO file_transports (name, transport, download_directory, upload_directory) VALUES ('Patron images (local)', 'local', '/tmp/patron-images', '/tmp/patron-images'); 3. Go to Tools > Patrons and circulation > Schedule patron image imports, click "New account" and enter: - Name: Staff photos - File transport: Patron images (local) - Active: checked - Field to use for patron matching: Card number then Save => The account is listed, with Active "Yes" 4. Pick three patrons that have no photo (A, B and C) and note their card numbers. In "ktd --shell", put any small image in a file (here sample.png) and make these files in /tmp/patron-images, with the card numbers as file names: cd /tmp/patron-images cp sample.png <card number of A>.png cp sample.png <card number of B>.png cp sample.png <card number of C>.png python3 -m zipfile -c batch.zip <card number of B>.png <card number of C>.png rm <card number of B>.png <card number of C>.png cp sample.png 99999999999.png (The zip holds the images of B and C, A has a loose image, and the last file matches no patron.) 5. From the Koha directory, do a test run: perl misc/cronjobs/patron_image_imports.pl => It says "Test run only" and lists the 3 files ("Processing: ..."). Nothing is imported: the files are still where they were, and no patron has a photo 6. Run it for real: perl misc/cronjobs/patron_image_imports.pl -c -v => (The files can be listed in any order) Processing: batch.zip 2 imported, 0 errors Processing: 99999999999.png PATRON_NOT_FOUND Processing: <card number of A>.png imported 7. Check the result: - Patrons A, B and C now have a photo on their patron record - In /tmp/patron-images, the imported files (batch.zip and A's image) have been moved to the "processed" directory and the file that did not match any patron is in the "error" directory - In "ktd --dbshell": SELECT patron_image_import_account_id, filename, status, details FROM patron_image_import_log; => 3 rows: batch.zip and A's image are "processed", 99999999999.png is an "error" (PATRON_NOT_FOUND) 8. Run the command of step 6 again => Nothing to process. Now copy sample.png to /tmp/patron-images/<card number of A>.png once more and run it again => "Already processed: <card number of A>.png": a file name is only processed once for an account, the file stays where it is and nothing is imported or logged again 9. Back on Tools > Schedule patron image imports, delete the account ("Delete", then "Yes, delete") => The account is gone, its rows in patron_image_import_log too, and the photos that were imported are still there 10. Run $ prove t/db_dependent/Koha/Patron/Image/Import.t \ t/db_dependent/Koha/PatronImageImportAccounts.t \ t/db_dependent/Koha/PatronImageImportLogs.t \ t/db_dependent/cronjobs/patron_image_imports.t => All pass Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #14 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209020 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209020&action=edit Bug 43638: Add 'Schedule patron image imports' page under Tools Add tools/schedule_patron_image_imports.pl, for staff with the tools > import_patrons permission, to manage the accounts polled by patron_image_imports.pl: list, add, edit and delete an account, with its name, its file transport, whether it is active, and the field used to match images to patrons (card number, username, or any patron attribute type with unique ID when ExtendedPatronAttributes is enabled). The page is linked from the Tools menu and from the Tools home page, under "Patrons and circulation". To test: 1. Go to Tools. Under "Patrons and circulation" there is "Schedule patron image imports" (it is also in the menu on the left); click it => "There are no scheduled patron image import accounts." 2. Click "New account", enter a name, choose a file transport (see the test plan of "Add patron_image_imports.pl cron job" if you have none), keep the account active and the matchpoint on "Card number", and Save => The account is listed with its ID, name, file transport, matchpoint and Active "Yes" 3. The "Edit" link of the account opens the same form 4. Delete the account: "Delete", then "Yes, delete" => The account is gone and the "There are no scheduled patron image import accounts." message is back Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 --- Comment #15 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209021 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209021&action=edit Bug 43638: Register patron_image_imports.pl in the default crontab templates Run the patron image imports once a day, at 4am, with the other default cron jobs: with a commented entry in misc/cronjobs/crontab.example, and, for the packages, in debian/koha-common.cron.daily, for each enabled instance with koha-foreach. It does nothing for an instance that has no patron image import account configured. To test: 1. Check that misc/cronjobs/crontab.example and debian/koha-common.cron.daily have the new "patron image imports" entry 2. See the test plan of "Add patron_image_imports.pl cron job" to run the job Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43638 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #16 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Andrew, thanks, and sorry for the empty commit messages: the patches had no description or test plan at all. I've re-attached all seven with a description each, and the full test plan is in comment 13 (the 'Add patron_image_imports.pl cron job' patch); the Tools page and crontab patches have short ones of their own. What the bug does: it adds scheduled, unattended import of patron photos. You set up one or more accounts under Tools > Schedule patron image imports (a file transport to poll, and the field to match patrons on: card number, username or a unique patron attribute), and a new cron job, misc/cronjobs/patron_image_imports.pl, picks up new zip archives and loose images from that transport and imports them with the engine from Bug 43637. Each file is then moved to a processed or error directory and logged once per account and file name, so the job is safe to run repeatedly. I ran the whole plan from scratch in KTD to make sure it works as written: a loose image and a zip of two images are imported (3 patrons get a photo), a file matching no patron goes to error/ with PATRON_NOT_FOUND, a test run (no -c) changes nothing, re-dropping an already processed file name is reported as 'Already processed' and left alone, and deleting the account removes its log rows but keeps the imported photos. The new tests (Image/Import.t, the two object classes and the cron job's own test, 22 tests) pass. Two things you'll want to know when testing: - The Administration > File transports form only offers FTP and SFTP, so unless you have such a server to hand the plan creates a 'local' transport with one SQL insert (there's a ready-made Koha::File::Transport::Local the cron job supports). - The Tools page has no view of the log: the outcome is in the cron job's -v output and the patron_image_import_log table. The patches are also rebased on current main (bug 43637's are needed first), and the schema update is regenerated with dbic --force, since a change to FileTransport.pm in main had made the generated footer stale. Setting the status back to Needs Signoff. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org