[Bug 27859] New: Marc export for search result
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Bug ID: 27859 Summary: Marc export for search result Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: glasklas@gmail.com This patch provides the option of exporting a search result as MARCXML or iso 2709. Ideally the export should be asynchronous but it was much trickier than first thought to fork and detach from Apache so gave that up for now. A link to the export is sent to the current user as an email as a simple way of delivering the response asynchronously (keeping it this way so don't have to refactor when/if the export is made asynchronous). -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #1 from David Gustafsson <glasklas@gmail.com> --- Created attachment 117718 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=117718&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Make sure the current user has an email under your control set 6) Perform a search 7) Export the serach result by choosing a format under the "Export results" drop down 8) Verify that link(s) with exported data has been mailed to the provided emial 9) Revoke the permission in 3) and make sure exporting is no longer possible 10) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #117718|0 |1 is obsolete| | --- Comment #2 from David Gustafsson <glasklas@gmail.com> --- Created attachment 117721 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=117721&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Make sure the current user has an email under your control set 6) Perform a search 7) Export the serach result by choosing a format under the "Export results" drop down 8) Verify that link(s) with exported data has been mailed to the provided emial 9) Revoke the permission in 3) and make sure exporting is no longer possible 10) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #3 from David Gustafsson <glasklas@gmail.com> --- When writing tests for this patch also discovered and fixed a bug in_array_to_marc($data) where $data is unintentionally modified by replacing the value of hash-ref in that strucure. $data is mutated when it should not which becomes apparent if calling _array_to_marc with the same argument multiple times. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Stefan Berndtsson <stefan.berndtsson@ub.gu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |stefan.berndtsson@ub.gu.se Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |glasklas@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #4 from David Gustafsson <glasklas@gmail.com> --- Created attachment 124892 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=124892&action=edit Use patron object instead of userenv -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #117721|0 |1 is obsolete| | Attachment #124892|0 |1 is obsolete| | --- Comment #5 from David Gustafsson <glasklas@gmail.com> --- Created attachment 124893 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=124893&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Make sure the current user has an email under your control set 6) Perform a search 7) Export the serach result by choosing a format under the "Export results" drop down 8) Verify that link(s) with exported data has been mailed to the provided emial 9) Revoke the permission in 3) and make sure exporting is no longer possible 10) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff --- Comment #6 from David Gustafsson <glasklas@gmail.com> --- Rebased against master. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #124893|0 |1 is obsolete| | --- Comment #7 from David Gustafsson <glasklas@gmail.com> --- Created attachment 128632 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128632&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Make sure the current user has an email under your control set 6) Perform a search 7) Export the serach result by choosing a format under the "Export results" drop down 8) Verify that link(s) with exported data has been mailed to the provided emial 9) Revoke the permission in 3) and make sure exporting is no longer possible 10) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Marc export for search |MARC export for search |result |results -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128632|0 |1 is obsolete| | --- Comment #8 from David Gustafsson <glasklas@gmail.com> --- Created attachment 131442 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=131442&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the syspref EnableSearchResultMARCExportEmail is enabled 5) Make sure the current user has the tools -> export_catalog permission 6) Make sure the current user has an email under your control set 7) Perform a search 8) Export the serach result by choosing a format under the "Export results" drop down 9) Verify that link(s) with exported data has been mailed to the provided email 10) Revoke the permission in 3) and make sure exporting is no longer possible 11) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #131442|0 |1 is obsolete| | --- Comment #9 from David Gustafsson <glasklas@gmail.com> --- Created attachment 131443 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=131443&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the syspref EnableSearchResultMARCExportEmail is enabled 5) Make sure the current user has the tools -> export_catalog permission 6) Make sure the current user has an email under your control set 7) Perform a search 8) Export the serach result by choosing a format under the "Export results" drop down 9) Verify that link(s) with exported data has been mailed to the provided email 10) Revoke the permission in 3) and make sure exporting is no longer possible 11) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply CC| |severine.queune@bulac.fr --- Comment #10 from Séverine Queune <severine.queune@bulac.fr> --- Sorry David, patch needs rebase :/ Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 27859: marc search result export Using index info to reconstruct a base tree... M Koha/SearchEngine/Elasticsearch/Search.pm M catalogue/search.pl M installer/data/mysql/mandatory/sysprefs.sql M koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt Falling back to patching base and 3-way merge... Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt Auto-merging installer/data/mysql/mandatory/sysprefs.sql Auto-merging catalogue/search.pl Auto-merging Koha/SearchEngine/Elasticsearch/Search.pm CONFLICT (content): Merge conflict in Koha/SearchEngine/Elasticsearch/Search.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 27859: marc search result export hint: Use 'git am --show-current-patch=diff' to see the failed patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #131443|0 |1 is obsolete| | --- Comment #11 from David Gustafsson <glasklas@gmail.com> --- Created attachment 132993 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132993&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the syspref EnableSearchResultMARCExportEmail is enabled 5) Make sure the current user has the tools -> export_catalog permission 6) Make sure the current user has an email under your control set 7) Perform a search 8) Export the serach result by choosing a format under the "Export results" drop down 9) Verify that link(s) with exported data has been mailed to the provided email 10) Revoke the permission in 3) and make sure exporting is no longer possible 11) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff --- Comment #12 from David Gustafsson <glasklas@gmail.com> --- Ok! Rebased against master. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132993|0 |1 is obsolete| | --- Comment #13 from David Gustafsson <glasklas@gmail.com> --- Created attachment 132997 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132997&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the syspref EnableSearchResultMARCExportEmail is enabled 5) Add a custom filter in the SearchResultMARCExportCustomFormats syspref, for example by copying one of the example formats 6) Make sure the current user has the tools -> export_catalog permission 7) Make sure the current user has an email under your control set 8) Perform a search 9) Export the search result for each format in the "Export all results" drop down 10) Verify that link(s) with exported data has been mailed to the provided email 11) Revoke the permission in 3) and make sure exporting is no longer possible 12) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132997|0 |1 is obsolete| | --- Comment #14 from David Gustafsson <glasklas@gmail.com> --- Created attachment 133076 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133076&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the syspref EnableSearchResultMARCExportEmail is enabled 5) Add a custom filter in the SearchResultMARCExportCustomFormats syspref, for example by copying one of the example formats 6) Make sure the current user has the tools -> export_catalog permission 7) Make sure the current user has an email under your control set 8) Perform a search 9) Export the search result for each format in the "Export all results" drop down 10) Verify that link(s) with exported data has been mailed to the provided email 11) Revoke the permission in 3) and make sure exporting is no longer possible 12) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #15 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi David, the QA test tools turn up some issues with this patch, can you please fix? 1) QA test tools FAIL Koha/SearchEngine/Elasticsearch.pm FAIL spelling prefered ==> preferred OK Koha/SearchEngine/Elasticsearch/Search.pm FAIL catalogue/search.pl FAIL critic # ControlStructures::ProhibitMutatingListFunctions: Got 1 violation(s). # Variables::ProhibitUnusedVariables: Got 1 violation(s). OK installer/data/mysql/atomicupdate/bug_27859-add_enable_search_result_marc_export_sysprefs.pl OK installer/data/mysql/mandatory/sysprefs.sql FAIL koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt FAIL filters wrong_html_filter at line 325 ( <li><a href="/cgi-bin/koha/catalogue/search.pl?count=[% results_per_page | uri %]&export=1&export_format=[% id %][% PROCESS sort_search_query %]">[% custom_export_formats.$id.name | html %]</a></li>) missing_filter at line 325 ( <li><a href="/cgi-bin/koha/catalogue/search.pl?count=[% results_per_page | uri %]&export=1&export_format=[% id %][% PROCESS sort_search_query %]">[% custom_export_formats.$id.name | html %]</a></li>) missing_filter at line 360 ( <div class="dialog message">[% export_message %]</div>) missing_filter at line 363 ( <div class="dialog error">[% export_error %]</div>) FAIL t/db_dependent/Koha/SearchEngine/Elasticsearch.t FAIL spelling indentical ==> identical indentical ==> identical 2) Translatability This will not be translatable and should be moved to templates, also the other places where export_links_html is set. HTML should not be generated in .pl files: + $export_links_html = $links_count > 1 ? + "<p>Some records exceeded the maximum size supported by ISO2709 and was exported as MARCXML instead.</p>" . $export_links_html : $export_links_html; Also here: $message = "<p>The export finished successfully:</p>" 3) System preferences I am a little in doubt about SearchResultMARCExportFromAddress. The from address can not always be set to be a library specific email, especially for installations hosted centrally by a support provider. We usually tend to use the reply-to for individual settings. But it should for sure be type = email in the .prefs file (activates validation). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133076|0 |1 is obsolete| | --- Comment #16 from David Gustafsson <glasklas@gmail.com> --- Created attachment 145243 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145243&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #17 from David Gustafsson <glasklas@gmail.com> --- Thanks for the review! Issues should now be fixed. Now using background job when performing the export so removed the feature for sending results in email. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145243|0 |1 is obsolete| | --- Comment #18 from David Gustafsson <glasklas@gmail.com> --- Created attachment 145244 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145244&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #19 from David Nind <david@davidnind.com> --- Just one item from testing. When I click on the "View detail of the enqueued job" (link is http://127.0.0.1:8081/cgi-bin/koha/admin/background_jobs.pl?op=view&id=1) I get an error trace page: Template process failed: file error - background_jobs/search_result_marc_export.inc: not found at /kohadevbox/koha/C4/Templates.pm line 127 in C4::Templates::output at /kohadevbox/koha/C4/Templates.pm line 127 124: 125: my $data; 126: binmode( STDOUT, ":encoding(UTF-8)" ); 127: $template->process( $self->filename, $vars, \$data ) 128: || die "Template process failed: ", $template->error(); 129: return $data; 130: } .... I get the same error if I try to view the job details from Administration > Jobs > Manage jobs -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145244|0 |1 is obsolete| | --- Comment #20 from David Gustafsson <glasklas@gmail.com> --- Created attachment 145263 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145263&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #21 from David Gustafsson <glasklas@gmail.com> --- (In reply to David Nind from comment #19)
Just one item from testing.
When I click on the "View detail of the enqueued job" (link is http://127.0.0.1:8081/cgi-bin/koha/admin/background_jobs.pl?op=view&id=1) I get an error trace page:
Template process failed: file error - background_jobs/search_result_marc_export.inc: not found at /kohadevbox/koha/C4/Templates.pm line 127
in C4::Templates::output at /kohadevbox/koha/C4/Templates.pm line 127
124: 125: my $data; 126: binmode( STDOUT, ":encoding(UTF-8)" ); 127: $template->process( $self->filename, $vars, \$data ) 128: || die "Template process failed: ", $template->error(); 129: return $data; 130: }
....
I get the same error if I try to view the job details from Administration > Jobs > Manage jobs
Ok! Forgot to include that file in the commit, have now fixed this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145263|0 |1 is obsolete| | --- Comment #22 from David Nind <david@davidnind.com> --- Created attachment 145300 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145300&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #23 from David Nind <david@davidnind.com> --- (In reply to David Gustafsson from comment #21)
Ok! Forgot to include that file in the commit, have now fixed this.
Thanks David! Now signed off. Tested with ES7. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact| |katrin.fischer@bsz-bw.de -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #24 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Testing... 1) QA test tools FAIL koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/search_result_marc_export.inc FAIL filters missing_filter at line 14 ( <li>[% format %]: <a href="[% report.export_links.$format | $raw %]">[% report.export_links.$format | html %]</a></li 2) Database update It would be better to use the new template for system preferences, but the output should include the names of the new preferences. 3) System preference definitions a) This needs to be 0/1 instead of yes/no: + - pref: EnableSearchResultMARCExport + type: boolean + default: yes + choices: + yes: Enable + no: Disable b) This doesn't read quite right: "A format have the required properties \"<strong>name</strong>\", \"<strong>fields</strong>\" and an optional \"<strong>multiple</strong>\".<br />" c) SearchResultMARCExportCustomFormats It says MARC export... but do we actually export MARC? Maybe a note where to find the filed names would help here. From the name, I had expected MARC fields. d) ElasticSearch should be Elasticsearch e) As this appears to be an Elasticserach only feature, all relating prefs should say so. Also you might want to add some links from one pref to the others involved. I got to stop here now since Elasticsearch doesn't want to cooperate in my koha-testing-docker environment right now :( -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145300|0 |1 is obsolete| | --- Comment #25 from David Gustafsson <glasklas@gmail.com> --- Created attachment 147927 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=147927&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #147927|0 |1 is obsolete| | --- Comment #26 from David Gustafsson <glasklas@gmail.com> --- Created attachment 147935 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=147935&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #147935|0 |1 is obsolete| | --- Comment #27 from David Gustafsson <glasklas@gmail.com> --- Created attachment 147936 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=147936&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #28 from David Gustafsson <glasklas@gmail.com> --- (In reply to Katrin Fischer from comment #24)
Testing...
1) QA test tools
FAIL koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/ search_result_marc_export.inc FAIL filters missing_filter at line 14 ( <li>[% format %]: <a href="[% report.export_links.$format | $raw %]">[% report.export_links.$format | html %]</a></li
2) Database update
It would be better to use the new template for system preferences, but the output should include the names of the new preferences.
3) System preference definitions
a) This needs to be 0/1 instead of yes/no: + - pref: EnableSearchResultMARCExport + type: boolean + default: yes + choices: + yes: Enable + no: Disable
b) This doesn't read quite right: "A format have the required properties \"<strong>name</strong>\", \"<strong>fields</strong>\" and an optional \"<strong>multiple</strong>\".<br />"
c) SearchResultMARCExportCustomFormats It says MARC export... but do we actually export MARC? Maybe a note where to find the filed names would help here. From the name, I had expected MARC fields.
d) ElasticSearch should be Elasticsearch
e) As this appears to be an Elasticserach only feature, all relating prefs should say so. Also you might want to add some links from one pref to the others involved.
I got to stop here now since Elasticsearch doesn't want to cooperate in my koha-testing-docker environment right now :(
Thanks for the feedback! The issues should now be fixed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=28793 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Sam Lau <samalau@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |samalau@gmail.com Status|Needs Signoff |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff --- Comment #29 from David Gustafsson <glasklas@gmail.com> --- Rebased against master. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Status|Needs Signoff |Failed QA --- Comment #30 from Lucas Gass <lucas@bywatersolutions.com> --- David, unfortunately your tests no longer apply cleanly. ( t/db_dependent/Koha/SearchEngine/Elasticsearch.t ) It looks like a simple fix where you just need to adjust the number of tests. QA Tool: FAIL koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt FAIL valid_template str/job_progess.inc: not found I don't see an include in Koha called this. Also, don't forget to tidy! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|katrin.fischer@bsz-bw.de | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #147936|0 |1 is obsolete| | --- Comment #31 from David Gustafsson <glasklas@gmail.com> --- Created attachment 162342 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=162342&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #32 from David Gustafsson <glasklas@gmail.com> --- The job_progress-files where misspelled in previous version of Koha, that's why the includes now fail and this is now fixed. The qa script complains about POD coverage being greater before for Koha/BackgroundJob/SearchResultExport.pm, but don't really understand why. It's the same as the other background job classes, and a coverage of 1 should be fine? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #162342|0 |1 is obsolete| | --- Comment #33 from David Gustafsson <glasklas@gmail.com> --- Created attachment 162343 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=162343&action=edit Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #34 from David Nind <david@davidnind.com> --- Created attachment 163761 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163761&action=edit Bug 27859: Export staff interface search results when using Elasticsearch Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the EnableElasticsearchSearchResultExport system preference is enabled 4) Make sure the current user has the export_catalog permission (Use all tools > Export bibliographic and holdings data) 5) Perform a search 6) Export the serach result by choosing a format under the "Export all results" drop down 7) Verify that link(s) with exported data appear on the background job detail page when the export is completed 8) Revoke the permission in step 4 and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #35 from David Nind <david@davidnind.com> --- Created attachment 163762 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=163762&action=edit Bug 27859: (follow-up) Update system preference descriptions Format and edit the descriptions for the new EnableElasticsearchSearchResultExport and ElasticsearchSearchResultExportCustomFormats system preferences. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #162343|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This enhancement enables release notes| |exporting search results | |from the staff interface | |when using Elasticsearch or | |Open Search. Two formats | |are available by default: | |MARC (UTF-8) and MARC XML. | |The export is run as a | |background job, with the | |results available for | |download from the details | |page for the completed job. | |Requires the export_catalog | |permission. | | | |Three new | |system preferences are used | |to configure exporting | |search results: | |- | |EnableElasticsearchSearchRe | |sultExport: enable the | |exporting of search results | | | |- | |ElasticsearchSearchResultEx | |portCustomFormats: define | |custom export formats | |- | |ElasticsearchSearchResultEx | |portLimit: limit the number | |of search results exported --- Comment #36 from David Nind <david@davidnind.com> --- I have: 1. Amended the title and test plan for the patch (to update to the new system preference name) 2. Added a follow-up with some edits to the system preference descriptions. Testing notes using KTD: 1. Start up KTD using Elasticsearch: ktd --es8 up 2. The new system preference EnableElasticsearchSearchResultExport is enabled by default. 3. Test exporting using custom export formats: . Test the example custom export formats in the ElasticsearchSearchResultExportCustomFormats system preference description -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com Status|Signed Off |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #163761|0 |1 is obsolete| | --- Comment #37 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178145 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178145&action=edit Bug 27859: Export staff interface search results when using Elasticsearch Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the EnableElasticsearchSearchResultExport system preference is enabled 4) Make sure the current user has the export_catalog permission (Use all tools > Export bibliographic and holdings data) 5) Perform a search 6) Export the serach result by choosing a format under the "Export all results" drop down 7) Verify that link(s) with exported data appear on the background job detail page when the export is completed 8) Revoke the permission in step 4 and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #163762|0 |1 is obsolete| | --- Comment #38 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178146 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178146&action=edit Bug 27859: (follow-up) Update system preference descriptions Format and edit the descriptions for the new EnableElasticsearchSearchResultExport and ElasticsearchSearchResultExportCustomFormats system preferences. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff --- Comment #39 from David Gustafsson <glasklas@gmail.com> --- Rebased against main. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #40 from David Cook <dcook@prosentient.com.au> --- What's the use case for this enhancement? Is the idea that staff do a search rather than a SQL report because it's easier for them to understand? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #41 from Stefan Berndtsson <stefan.berndtsson@ub.gu.se> ---
Is the idea that staff do a search rather than a SQL report because it's easier for them to understand?
Search is more accessible to the average user (not everyone can create a report). Reports that fetch subfield content from MARC datafields are often VERY slow, where the Elasticsearch index is a lot faster and less demanding on the database and the system. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #42 from David Cook <dcook@prosentient.com.au> --- (In reply to Stefan Berndtsson from comment #41)
Is the idea that staff do a search rather than a SQL report because it's easier for them to understand?
Search is more accessible to the average user (not everyone can create a report). Reports that fetch subfield content from MARC datafields are often VERY slow, where the Elasticsearch index is a lot faster and less demanding on the database and the system.
Thanks for that, Stefan. I figured that would be the use case, but I wanted to make sure. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #43 from David Cook <dcook@prosentient.com.au> --- I think that there's a number of other areas where we've been interested in the idea of doing background processing and presenting the user with a link to a download, so this is interesting. It looks like these files will need to manually cleaned up at the moment? Could a method be added to cleanup_db.pl to automatically clean them up after a certain amount of time? I do think that librarians would like this feature. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #44 from David Cook <dcook@prosentient.com.au> --- Comment on attachment 178145 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178145 Bug 27859: Export staff interface search results when using Elasticsearch Review of attachment 178145: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=27859&attachment=178145) ----------------------------------------------------------------- ::: installer/data/mysql/atomicupdate/bug_27859-add_enable_search_result_marc_export_sysprefs.pl @@ +16,5 @@
+ $dbh->do(q{ INSERT IGNORE INTO systempreferences (variable, value, options, explanation, type) VALUES ('ElasticsearchSearchResultExportLimit', NULL, NULL, 'Search result export limit', 'integer') }); + say $out "Added new system preference 'ElasticsearchSearchResultExportLimit'"; + + $dbh->do(q{ UPDATE systempreferences SET options = 'base64ISO2709|ARRAY' WHERE variable = 'ElasticsearchMARCFormat' }); + $dbh->do(q{ UPDATE systempreferences SET value = 'base64ISO2709' WHERE variable = 'ElasticsearchMARCFormat' AND value = 'ISO2709' });
I like the idea of changing ISO2709 to base64ISO2709, as it's annoying to translate between base64ISO2709 and ISO2709 (whereas ARRAY and MARCXML can be matched easily). That said... that seems like a different bug report to me. It'll also conflict with bug 28370 if it gets accepted, so we'll want to be careful there. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32278 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=36272 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178145|0 |1 is obsolete| | --- Comment #45 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178370 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178370&action=edit Bug 27859: Export staff interface search results when using Elasticsearch Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the EnableElasticsearchSearchResultExport system preference is enabled 4) Make sure the current user has the export_catalog permission (Use all tools > Export bibliographic and holdings data) 5) Perform a search 6) Export the serach result by choosing a format under the "Export all results" drop down 7) Verify that link(s) with exported data appear on the background job detail page when the export is completed 8) Revoke the permission in step 4 and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178146|0 |1 is obsolete| | --- Comment #46 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178371 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178371&action=edit Bug 27859: (follow-up) Update system preference descriptions Format and edit the descriptions for the new EnableElasticsearchSearchResultExport and ElasticsearchSearchResultExportCustomFormats system preferences. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff --- Comment #47 from David Gustafsson <glasklas@gmail.com> --- (In reply to David Cook from comment #44)
Comment on attachment 178145 [details] [review] Bug 27859: Export staff interface search results when using Elasticsearch
Review of attachment 178145 [details] [review]: -----------------------------------------------------------------
::: installer/data/mysql/atomicupdate/bug_27859- add_enable_search_result_marc_export_sysprefs.pl @@ +16,5 @@
+ $dbh->do(q{ INSERT IGNORE INTO systempreferences (variable, value, options, explanation, type) VALUES ('ElasticsearchSearchResultExportLimit', NULL, NULL, 'Search result export limit', 'integer') }); + say $out "Added new system preference 'ElasticsearchSearchResultExportLimit'"; + + $dbh->do(q{ UPDATE systempreferences SET options = 'base64ISO2709|ARRAY' WHERE variable = 'ElasticsearchMARCFormat' }); + $dbh->do(q{ UPDATE systempreferences SET value = 'base64ISO2709' WHERE variable = 'ElasticsearchMARCFormat' AND value = 'ISO2709' });
I like the idea of changing ISO2709 to base64ISO2709, as it's annoying to translate between base64ISO2709 and ISO2709 (whereas ARRAY and MARCXML can be matched easily). That said... that seems like a different bug report to me. It'll also conflict with bug 28370 if it gets accepted, so we'll want to be careful there.
Did you link to the wrong bug? Seems it has been pushed and is not related. I agree it's not ideal to have the format name change as part of this bug, I just hoped I could sneak it in I suppose. Would be pretty simple to break it out I suppose. The syspref value actually was base64ISO2709 in bug 19893 before it was removed since I believe we didn't want to allow the user to select a less performant format, but was reintroduced in bug 22258 as ISO2709 when the ARRAY format was added. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #48 from David Gustafsson <glasklas@gmail.com> --- I might also have added the format name change to this bug as it makes more sense in this context. In the original code the value is never checked, it just defaults to base64ISO2709 if not set to ARRAY, so I guess without the refactoring of the record encoding method the change doesn't make as much sense. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |39171 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39171 [Bug 39171] Rename IS02709 ElasticsearchMARCFormat to base64ISO2709 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |BLOCKED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178370|0 |1 is obsolete| | --- Comment #49 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178376 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178376&action=edit Bug 27859: Export staff interface search results when using Elasticsearch Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the EnableElasticsearchSearchResultExport system preference is enabled 4) Make sure the current user has the export_catalog permission (Use all tools > Export bibliographic and holdings data) 5) Perform a search 6) Export the serach result by choosing a format under the "Export all results" drop down 7) Verify that link(s) with exported data appear on the background job detail page when the export is completed 8) Revoke the permission in step 4 and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178371|0 |1 is obsolete| | --- Comment #50 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178377 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178377&action=edit Bug 27859: (follow-up) Update system preference descriptions Format and edit the descriptions for the new EnableElasticsearchSearchResultExport and ElasticsearchSearchResultExportCustomFormats system preferences. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #51 from David Gustafsson <glasklas@gmail.com> --- Now broken out the format change into bug 39171. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|BLOCKED |Failed QA --- Comment #52 from David Cook <dcook@prosentient.com.au> --- Failing QA for a couple of things. 1. New features like EnableElasticsearchSearchResultExport should default to off, but it's currently defaulting to on. 2. In "Export all results", "MARC XML" should be "MARCXML" to be consistent with other interfaces. 3. I'm following the test plan but it keeps saying Failed with this report: "Search results export for the query " failed with the following errors:" -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #53 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178399 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178399&action=edit Bug 27859: Fix background job error and js progress callback -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #54 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178400 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178400&action=edit Bug 27859: Don't enable feature by default -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178399|0 |1 is obsolete| | --- Comment #55 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178401 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178401&action=edit Bug 27859: Fix background job error and js progress callback -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178400|0 |1 is obsolete| | --- Comment #56 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178402 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178402&action=edit Bug 27859: Don't enable feature by default -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178401|0 |1 is obsolete| | --- Comment #57 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178406 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178406&action=edit Bug 27859: (follow up) Fix background job error and js progress callback -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #58 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178407 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178407&action=edit Bug 27859: (follow-up) Don't enable feature by default -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #59 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178408 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178408&action=edit Bug 27859: (follow-up) Correct MARC XML to MARCXML -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178402|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #60 from David Gustafsson <glasklas@gmail.com> --- (In reply to David Cook from comment #52)
Failing QA for a couple of things.
1. New features like EnableElasticsearchSearchResultExport should default to off, but it's currently defaulting to on.
Ok, didn't know about that. This has now been fixed.
2. In "Export all results", "MARC XML" should be "MARCXML" to be consistent with other interfaces.
Yes, that\s clearly wrong. Fixed there and on the syspref page.
3. I'm following the test plan but it keeps saying Failed with this report:
"Search results export for the query " failed with the following errors:"
My mistake, should have tested it again after the rabase. Had some issues with running scheduled jobs in my environment so that's why, but made sure to fix this to be able to test. The error was caused by my changing that the reture value of search_documents_encode is no longer binary but a perl utf8 string, so mod5_hex fails, so need to encode the data before calculating a hash. There has also been some changes to the previous callback arguments of the updateProgress javascript function, which I also had to be adjusted for. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Esther Melander <esther.melander@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |esther.melander@bywatersolu | |tions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Roman Dolny <roman.dolny@jezuici.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA CC| |roman.dolny@jezuici.pl --- Comment #61 from Roman Dolny <roman.dolny@jezuici.pl> --- What about ElasticsearchSearchResultExportLimit system preference? I can't confirm it works - set it, but it has no impact for limit number of exported records. BTW, shouldn't we change "items" to "records" (or "elements"?) in description ("...search result items (Elasticsearch only).")? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178376|0 |1 is obsolete| | --- Comment #62 from David Gustafsson <glasklas@gmail.com> --- Created attachment 189760 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189760&action=edit Bug 27859: Export staff interface search results when using Elasticsearch Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the EnableElasticsearchSearchResultExport system preference is enabled 4) Make sure the current user has the export_catalog permission (Use all tools > Export bibliographic and holdings data) 5) Perform a search 6) Export the serach result by choosing a format under the "Export all results" drop down 7) Verify that link(s) with exported data appear on the background job detail page when the export is completed 8) Revoke the permission in step 4 and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178377|0 |1 is obsolete| | --- Comment #63 from David Gustafsson <glasklas@gmail.com> --- Created attachment 189761 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189761&action=edit Bug 27859: (follow-up) Update system preference descriptions Format and edit the descriptions for the new EnableElasticsearchSearchResultExport and ElasticsearchSearchResultExportCustomFormats system preferences. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178406|0 |1 is obsolete| | --- Comment #64 from David Gustafsson <glasklas@gmail.com> --- Created attachment 189762 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189762&action=edit Bug 27859: (follow up) Fix background job error and js progress callback -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178407|0 |1 is obsolete| | --- Comment #65 from David Gustafsson <glasklas@gmail.com> --- Created attachment 189763 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189763&action=edit Bug 27859: (follow-up) Don't enable feature by default -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178408|0 |1 is obsolete| | --- Comment #66 from David Gustafsson <glasklas@gmail.com> --- Created attachment 189764 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189764&action=edit Bug 27859: (follow-up) Correct MARC XML to MARCXML -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 --- Comment #67 from David Gustafsson <glasklas@gmail.com> --- Created attachment 189765 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189765&action=edit Bug 27859: (follow-up) Fix search export limit syspref name -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #68 from David Gustafsson <glasklas@gmail.com> --- @Roman Dolny I think that syspref was renamed and the old name was still used in the code, this has now been fixed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27859 Chris Mathevet <chris.mathevet@inLibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |chris.mathevet@inLibro.com Status|Needs Signoff |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org