[Bug 43603] New: Add batch scan import for article requests (folder/SFTP)
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 Bug ID: 43603 Summary: Add batch scan import for article requests (folder/SFTP) Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: martin.renvoize@openfifth.co.uk QA Contact: testopia@bugs.koha-community.org CC: jake.deery@openfifth.co.uk, robin@catalyst.net.nz Target Milestone: --- This patch adds a batch import mechanism that automatically completes scanned article requests by polling a configured file transport (local folder, SFTP, or FTP, reusing the existing Koha::File::Transport infrastructure) for zip archives. Each zip archive is expected to contain scanned files plus an idlink.txt (or datalink.txt) mapping file, in the same convention used by tools/picture-upload.pl and tools/upload-cover-image.pl. Each mapped row is matched against a PROCESSING/SCAN Koha::ArticleRequest, the file is registered under Koha's existing Upload feature, and the request is marked completed. This is part 2 of a two-part split of the original bug 20570 ("Add script to complete scanned article requests"), which now covers only the REST API for editing/completing article requests. This bug depends on bug 20570 landing first, since it builds on the REST API work and retires the 2018-era match_scans/search_scans_in_process methods and misc/migration_tools/complete_article_requests.pl script that bug 20570 deliberately left in place. Adds: - Two new tables: article_request_scan_accounts (config, references file_transports) and article_request_scan_log (idempotency tracking) - Koha::ArticleRequestScanAccount(s) and Koha::ArticleRequestScanLog(s) object classes - admin/article_request_scan_accounts.pl, a CRUD page for scan accounts - Koha::ArticleRequest::ScanImport, the zip-unpack-and-match engine - misc/cronjobs/article_request_scans.pl, replacing misc/migration_tools/complete_article_requests.pl Test plan: 1. Apply this patch (depends on bug 20570) 2. Configure a file transport (admin/file_transports.pl) and a scan account (admin/article_request_scan_accounts.pl) referencing it 3. Build a zip containing a scanned file plus an idlink.txt mapping it to a PROCESSING/SCAN article request 4. Run misc/cronjobs/article_request_scans.pl -c -v and confirm the request completes with a working urls value 5. Run the new test suites: prove t/db_dependent/Koha/ArticleRequestScanAccounts.t t/db_dependent/Koha/ArticleRequestScanLogs.t t/db_dependent/Koha/ArticleRequest/ScanImport.t -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |20570 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20570 [Bug 20570] Add script to complete scanned article requests -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Sponsorship status|--- |Sponsored Patch complexity|--- |Medium patch Status|NEW |Needs Signoff Assignee|koha-bugs@lists.koha-commun |martin.renvoize@openfifth.c |ity.org |o.uk -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 --- Comment #1 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206589 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206589&action=edit Bug 43603: Add article_request_scan_accounts and article_request_scan_log tables Adds two new tables backing the batch scan import for article requests: - article_request_scan_accounts: one row per configured scan source, referencing an existing file_transports row, plus the upload authorised value category used to register retrieved scan files and an active flag controlling whether article_request_scans.pl polls it. - article_request_scan_log: idempotency tracking, recording the outcome for every filename already processed per account so it is never reprocessed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 --- Comment #2 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206590 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206590&action=edit Bug 43603: Automated Schema Update -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 --- Comment #3 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206591 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206591&action=edit Bug 43603: Add relationships and boolean flag to article_request_scan schema Adds the is_boolean flag for the active column on Koha::Schema::Result::ArticleRequestScanAccount, matching the tinyint_has_boolean_flag QA convention. The belongs_to relationships on both new Result classes (file_transport and article_request_scan_account) were already correctly auto-detected by dbic --force and required no manual edit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 --- Comment #4 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206592 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206592&action=edit Bug 43603: Add Koha::ArticleRequestScanAccount(s) and Koha::ArticleRequestScanLog(s) Adds the object classes for the two new tables: - Koha::ArticleRequestScanAccount(s): scan source configuration. Its file_transport accessor resolves via Koha::File::Transports->find() rather than blessing the related row through the singular Koha::File::Transport base class, so the returned object dispatches to its concrete backend subclass (::Local, ::FTP, ::SFTP) - mirroring the existing pattern in Koha::Edifact::Transport for the same file_transport_id relationship. Blessing via the polymorphic base class would return an object whose connect/download_file/list_files methods are all unimplemented stubs. - Koha::ArticleRequestScanLog(s): per-account, per-filename idempotency records consulted by article_request_scans.pl before reprocessing a file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206593 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206593&action=edit Bug 43603: Add admin page for article request scan accounts Adds a plain CGI+Template-Toolkit admin page for staff to configure article request scan accounts, each referencing an existing Koha::File::Transports row. Closely modeled on admin/marc_order_accounts.pl/.tt, using the Koha::ArticleRequestScanAccount(s) object classes added in a previous commit. Linked from the Administration home page alongside the existing File transports entry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 --- Comment #6 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206594 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206594&action=edit Bug 43603: Add Koha::ArticleRequest::ScanImport Adds the matching engine that consumes a batch scan-return zip archive: locates its idlink.txt/datalink.txt mapping file, and for each mapped row finds the matching PROCESSING/SCAN Koha::ArticleRequest, registers the scanned file under Koha's existing Upload feature (building the Koha::UploadedFile row directly, since Koha::Uploader is hard-wired to a CGI streaming callback and can't be reused for a file already on local disk), sets the request's urls, and completes it. Row-level failures (no such request, wrong status/format, missing file, unparsable row, path traversal, or any other row-scoped exception) are accumulated in the returned errors array and do not stop processing of the remaining rows. Only zip-level failures (unreadable archive, no mapping file found) die. Path traversal is guarded structurally: any mapped row or zip member name containing a ".." path component is rejected before ever touching the filesystem, plus a realpath-based containment check once a file is known to exist (also covers symlink escapes). A naive post-hoc "does the resolved path still start with the base directory" string check is not sufficient here - on Unix, File::Spec's rel2abs/canonpath do not collapse ".." segments, so that check would never actually catch a "../../etc/passwd"-style row. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 --- Comment #7 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 206595 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206595&action=edit Bug 43603: Add article_request_scans.pl cron job Polls every active article_request_scan_accounts row's configured Koha::File::Transport for new .zip files, matches each one's idlink.txt/datalink.txt mapping against PROCESSING/SCAN article requests via Koha::ArticleRequest::ScanImport, and archives processed/failed zips into processed/ and error/ subdirectories. Already-processed filenames (per account, via article_request_scan_log) are never reprocessed, and the idempotency check happens before download. Archive-move failures are logged rather than left silent, so a misconfigured or unwritable transport shows up in the cron log instead of just quietly leaving a zip in place. For a Local transport, processed/ and error/ are created under the configured download_directory before archiving into them (in confirm mode only): Local's rename_file() is a plain File::Copy::move(), which does not create missing destination directories on its own. This replaces misc/migration_tools/complete_article_requests.pl (the 2018-era match_scans/search_scans_in_process script), which bug 20570 deliberately left in place while it reworked the REST API side of article request completion. This bug depends on bug 20570 for that REST API work. Test plan: 1. Apply this patch series 2. Configure a file transport (admin/file_transports.pl) and a scan account (admin/article_request_scan_accounts.pl) referencing it - note the new "Article request scan accounts" link on the Administration home page 3. Build a zip containing a scanned file plus an idlink.txt mapping it to a PROCESSING/SCAN article request 4. Run misc/cronjobs/article_request_scans.pl -c -v and confirm the request completes with a working urls value, and the zip is moved into the transport's processed/ directory 5. Repeat with a zip mapping to a nonexistent or already-COMPLETED request and confirm it is reported as an error, archived into error/, and logged to article_request_scan_log without stopping processing of other accounts 6. Run the new test suites: prove t/db_dependent/Koha/ArticleRequestScanAccounts.t \ t/db_dependent/Koha/ArticleRequestScanLogs.t \ t/db_dependent/Koha/ArticleRequest/ScanImport.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 --- Comment #8 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- This is an early proof of work.. I'm not 100% sure about the Log table and classes (We could just set somewhere in the configuration page to archive (move) or delete processed files instead.. we don't really output errors anywhere especially. I also followed the idlink.txt format of other bulk load systems we have (Patron Images and Cover Images), but again, I'm open to opinions on that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 --- Comment #9 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- This is meant as an alternative for bug 20570 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=20570 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43603 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|20570 |20472 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20472 [Bug 20472] Add digital scan as optional format to Article Requests https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20570 [Bug 20570] Add script to complete scanned article requests -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org