[Bug 42882] New: Add option to disable setting time and permissions for File Transports
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Bug ID: 42882 Summary: Add option to disable setting time and permissions for File Transports Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: kyle@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org Target Milestone: --- Some vendors don't allow the SFTP use to change the time and permissions on uploaded files. This results in the error "Couldn't setstat remote file: General failure at /usr/share/koha/lib/Koha/File/Transport/SFTP.pm line 97." for SFTP transports. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #1 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- We should add the option to File Transports to disable setting the time and permissions. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |kyle@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #2 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200718 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200718&action=edit Bug 42882: Add copy_file_attrs column to file_transports Patch from commit 8ce6001 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #3 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200719 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200719&action=edit Bug 42882: Honor copy_file_attrs in SFTP uploads By default Net::SFTP::Foreign copies the local file's permissions and timestamps onto the remote file at the end of an upload, using an SFTP setstat operation. Some SFTP servers reject setstat, and the upload then fails with an error like: ERROR: Couldn't setstat remote file: General failure at Koha/File/Transport/SFTP.pm line 97. even though the file contents transferred correctly. This breaks EDIFACT order uploads ( edi_cron.pl ) and any other SFTP upload to such a server. This patch makes Koha::File::Transport::SFTP pass copy_perm and copy_time to put() based on the new copy_file_attrs setting. When copy_file_attrs is off, the setstat is skipped and the upload succeeds. FTP and local transports don't perform a setstat, so the setting only affects SFTP. Test Plan: 1) Apply this patch 2) Run updatedatabase.pl 3) Restart all the things! 4) prove t/db_dependent/Koha/File/Transport/SFTP.t 5) Note the tests pass! 6) Go to Administration -> File transports and add or edit an SFTP transport 7) Note the new "Copy file attributes" option, set to Enabled by default 8) Change Transport to FTP, note the option is disabled ( as with passive mode ) 9) Set Transport back to SFTP, set the option to Disabled and save, note it is kept -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #4 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200720 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200720&action=edit Bug 42882: Add copy_file_attrs to the file transports REST API Patch from commit 6052338 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #5 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200721 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200721&action=edit Bug 42882: Add copy_file_attrs to the file transports editor Patch from commit a54f3c0 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #6 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200722 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200722&action=edit Bug 42882: Add unit tests Patch from commit 7f31968 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #7 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200723 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200723&action=edit Bug 42882: Update Schema files [DO NOT PUSH] Patch from commit 67e3aae -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA CC| |david@davidnind.com --- Comment #8 from David Nind <david@davidnind.com> --- After the patches, updatedatabase, yarn build, restart, etc., I'm getting an Error 500 on the file transports page, and it sits with "Processing...": jquery-3.6.0.min_26.0600003.js:2 GET http://127.0.0.1:8081/api/v1/config/file_transports?_page=1&_per_page=20&_match=contains&_order_by=%2Bme.name 500 (Internal Server Error) datatables_26.0600003.js:413 Got 500 (Internal Server Error) datatables_26.0600003.js:436 DataTables warning: table id=file_transports - Ajax error. For more information about this error, please see https://datatables.net/tn/7 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200718|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200719|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200720|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200721|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200722|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200723|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #9 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 203597 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203597&action=edit Bug 42882: Add copy_file_attrs column to file_transports Patch from commit b39aa3e -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #10 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 203598 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203598&action=edit Bug 42882: Honor copy_file_attrs in SFTP uploads By default Net::SFTP::Foreign copies the local file's permissions and timestamps onto the remote file at the end of an upload, using an SFTP setstat operation. Some SFTP servers reject setstat, and the upload then fails with an error like: ERROR: Couldn't setstat remote file: General failure at Koha/File/Transport/SFTP.pm line 97. even though the file contents transferred correctly. This breaks EDIFACT order uploads ( edi_cron.pl ) and any other SFTP upload to such a server. This patch makes Koha::File::Transport::SFTP pass copy_perm and copy_time to put() based on the new copy_file_attrs setting. When copy_file_attrs is off, the setstat is skipped and the upload succeeds. FTP and local transports don't perform a setstat, so the setting only affects SFTP. Test Plan: 1) Apply this patch 2) Run updatedatabase.pl 3) Restart all the things! 4) prove t/db_dependent/Koha/File/Transport/SFTP.t 5) Note the tests pass! 6) Go to Administration -> File transports and add or edit an SFTP transport 7) Note the new "Copy file attributes" option, set to Enabled by default 8) Change Transport to FTP, note the option is disabled ( as with passive mode ) 9) Set Transport back to SFTP, set the option to Disabled and save, note it is kept -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #11 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 203599 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203599&action=edit Bug 42882: Add copy_file_attrs to the file transports REST API Patch from commit 5a836da -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #12 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 203600 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203600&action=edit Bug 42882: Add copy_file_attrs to the file transports editor Patch from commit 2cbb688 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #13 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 203601 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203601&action=edit Bug 42882: Add unit tests Patch from commit 8e996b5 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #14 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 203602 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203602&action=edit Bug 42882: (QA follow-up) File transport tests leave rows in the database Several subtests in t/db_dependent/Koha/File/Transport/SFTP.t and t/db_dependent/Koha/File/Transport/FTP.t call $builder->build_object( { class => 'Koha::File::Transports' } ) without wrapping the subtest in a transaction, so the rows are still there after the test finishes. TestBuilder fills every column it wasn't given a value for, nullable ones included, so each leftover row gets a random string in status. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #15 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 203603 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203603&action=edit Bug 42882: Update Schema files [DO NOT PUSH] Patch from commit 0f2a251 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #203597|0 |1 is obsolete| | Attachment #203598|0 |1 is obsolete| | Attachment #203599|0 |1 is obsolete| | Attachment #203600|0 |1 is obsolete| | Attachment #203601|0 |1 is obsolete| | Attachment #203602|0 |1 is obsolete| | Attachment #203603|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #16 from David Nind <david@davidnind.com> --- Created attachment 203662 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203662&action=edit Bug 42882: Add copy_file_attrs column to file_transports Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #17 from David Nind <david@davidnind.com> --- Created attachment 203663 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203663&action=edit Bug 42882: Honor copy_file_attrs in SFTP uploads By default Net::SFTP::Foreign copies the local file's permissions and timestamps onto the remote file at the end of an upload, using an SFTP setstat operation. Some SFTP servers reject setstat, and the upload then fails with an error like: ERROR: Couldn't setstat remote file: General failure at Koha/File/Transport/SFTP.pm line 97. even though the file contents transferred correctly. This breaks EDIFACT order uploads ( edi_cron.pl ) and any other SFTP upload to such a server. This patch makes Koha::File::Transport::SFTP pass copy_perm and copy_time to put() based on the new copy_file_attrs setting. When copy_file_attrs is off, the setstat is skipped and the upload succeeds. FTP and local transports don't perform a setstat, so the setting only affects SFTP. Test Plan: 1) Apply this patch 2) Run updatedatabase.pl 3) Restart all the things! 4) prove t/db_dependent/Koha/File/Transport/SFTP.t 5) Note the tests pass! 6) Go to Administration -> File transports and add or edit an SFTP transport 7) Note the new "Copy file attributes" option, set to Enabled by default 8) Change Transport to FTP, note the option is disabled ( as with passive mode ) 9) Set Transport back to SFTP, set the option to Disabled and save, note it is kept Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #18 from David Nind <david@davidnind.com> --- Created attachment 203664 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203664&action=edit Bug 42882: Add copy_file_attrs to the file transports REST API Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #19 from David Nind <david@davidnind.com> --- Created attachment 203665 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203665&action=edit Bug 42882: Add copy_file_attrs to the file transports editor Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #20 from David Nind <david@davidnind.com> --- Created attachment 203666 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203666&action=edit Bug 42882: Add unit tests Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #21 from David Nind <david@davidnind.com> --- Created attachment 203667 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203667&action=edit Bug 42882: (QA follow-up) File transport tests leave rows in the database Several subtests in t/db_dependent/Koha/File/Transport/SFTP.t and t/db_dependent/Koha/File/Transport/FTP.t call $builder->build_object( { class => 'Koha::File::Transports' } ) without wrapping the subtest in a transaction, so the rows are still there after the test finishes. TestBuilder fills every column it wasn't given a value for, nullable ones included, so each leftover row gets a random string in status. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #22 from David Nind <david@davidnind.com> --- Created attachment 203668 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203668&action=edit Bug 42882: Update Schema files [DO NOT PUSH] Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 --- Comment #23 from David Nind <david@davidnind.com> --- I've tested, but without access to any real FTP or SFTP servers (so the test connctions eventually fail). I'm not sure what is actually required, but I did an updatedatabase, yarn build and dbic (although I get an error). If I don't do these I get errors with the page sitting on loading after attempting to add an SFTP or FTP server: Something went wrong when loading the table. 500: Internal Server error Properties not allowed: copy_file_attrs. ... (repeated a couple of times) With running dbic I get this after it completes: ... Dumping manual schema for Koha::Schema to directory ./ ... DBIx::Class::Schema::Loader::make_schema_at(): Checksum mismatch in './/Koha/Schema/Result/Itemtype.pm', the auto-generated part of the file has been modified outside of this loader. Aborting. If you want to overwrite these modifications, set the 'overwrite_modifications' loader option. The tests pass 8-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |tomascohen@gmail.com |y.org | CC| |lisette@bywatersolutions.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42882 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|tomascohen@gmail.com |jacob.omara@openfifth.co.uk -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org