https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42032 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42032 [Bug 42032] Add diff support to AUTHORITIES action logs -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |martin.renvoize@openfifth.c |ity.org |o.uk -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #1 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 196219 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196219&action=edit Bug 42188: Use MARC-in-JSON for biblio action log diffs Replace the custom _marc_record_to_diffable helper with MARC::Record::MiJ's to_mij_structure method, which produces standard MARC-in-JSON. This fixes three problems with the old approach: - Dollar signs used as subfield delimiters in the formatted string were ambiguous when subfield content itself contained " $x " sequences. - Grouping fields by tag into a hash lost inter-field ordering, making lossless reconstruction of the MARC record impossible. - Non-ASCII characters were subject to Perl string context encoding issues; JSON serialisation of the MiJ structure handles UTF-8 cleanly. _unblessed_for_log in Koha::Biblio is updated to return only the three biblio columns that are genuinely independent of the MARC record (frameworkcode, opac_suppressed, serial). The remaining columns (author, title, subtitle, notes, etc.) are denormalised copies of MARC data already captured in the _marc key, so including them produced redundant and noisy diffs. The fmt() helper in viewlog.js is updated to render JavaScript arrays as ordered lists rather than objects with numeric keys, improving the display of MARC subfield arrays in the diff table. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #3 from Janusz Kaczmarek <januszop@gmail.com> --- Martin, and how about modifying (in C4::Log.pm line 99): $infos = "biblio " . Dumper( ref $original eq 'HASH' ? $original : {} ); into: $infos = "biblio " . ( ref $original eq 'HASH' ? encode_json($original) : qw{} ); or even better: $infos = "biblio " . ( ref $original eq 'HASH' ? to_json($original, {utf8 => 1, pretty => 1}) : qw{} ); This would solve the problem of non-ASCII characters and make the Info column more readable, IMO. And would not change the diff. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #4 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 196444 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196444&action=edit Bug 42188: Use to_json for biblio action log info column Replace Dumper with to_json for the biblio info column in logaction, fixing UTF-8 encoding and extending coverage to ADD and DELETE actions which previously logged only the literal string "biblio". All three actions (ADD, MODIFY, DELETE) now log "biblio " followed by a pretty-printed JSON representation of the relevant state: pre-change for MODIFY, new state for ADD, and deleted state for DELETE. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 196445 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196445&action=edit Bug 42188: Use MARC-in-JSON for biblio action log diffs Replace the custom _marc_record_to_diffable helper with MARC::Record::MiJ's to_mij_structure method, which produces standard MARC-in-JSON. This fixes three problems with the old approach: - Dollar signs used as subfield delimiters in the formatted string were ambiguous when subfield content itself contained " $x " sequences. - Grouping fields by tag into a hash lost inter-field ordering, making lossless reconstruction of the MARC record impossible. - Non-ASCII characters were subject to Perl string context encoding issues; JSON serialisation of the MiJ structure handles UTF-8 cleanly. _unblessed_for_log in Koha::Biblio is updated to return only the three biblio columns that are genuinely independent of the MARC record (frameworkcode, opac_suppressed, serial). The remaining columns (author, title, subtitle, notes, etc.) are denormalised copies of MARC data already captured in the _marc key, so including them produced redundant and noisy diffs. The fmt() helper in viewlog.js is updated to render JavaScript arrays as ordered lists rather than objects with numeric keys, improving the display of MARC subfield arrays in the diff table. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #7 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 196447 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196447&action=edit Bug 42188: Use to_json for biblio action log info column Replace Dumper with to_json for the biblio info column in logaction, fixing UTF-8 encoding and extending coverage to ADD and DELETE actions which previously logged only the literal string "biblio". All three actions (ADD, MODIFY, DELETE) now log "biblio " followed by a pretty-printed JSON representation of the relevant state: pre-change for MODIFY, new state for ADD, and deleted state for DELETE. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #196445|0 |1 is obsolete| | Attachment #196446|0 |1 is obsolete| | Attachment #196447|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #10 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 197923 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=197923&action=edit Bug 42188: Add/update unit tests for MARC-in-JSON action log diffs Update the existing _unblessed_for_log() subtest to reflect the new behaviour: MARC-derived columns (title, author, copyrightdate, etc.) are now excluded, while frameworkcode is always included even when it is an empty string (the Default framework). Add a new subtest 'CataloguingLog MARC-in-JSON diff tests' covering: - ADD: diff _marc key is MARC-in-JSON (has leader + fields array), indicators and subfield values are correct, and MARC-derived biblio columns are absent from the diff. - MODIFY: diff _marc key present, MARC-derived columns absent. - DELETE: diff _marc key present with correct MARC-in-JSON structure. Use from_json() instead of decode_json() when reading diffs back from the database, as the DBD layer returns decoded Perl Unicode strings which decode_json (which expects UTF-8 bytes) would reject for any non-ASCII content. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #12 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 197925 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=197925&action=edit Bug 42188: (follow-up) Skip JSON payload in info column on ADD The info column for CATALOGUING action logs is, by convention, the record state before modification -- MODIFY writes the pre-change JSON payload, DELETE writes the state at time of deletion, but ADD has no pre-change state. Writing the new record state into info on ADD duplicates the state that the subsequent MODIFY (if any) captures as its "before" snapshot, and conflicts with that convention. Skip the JSON payload on ADD so info degrades to the bare "biblio" prefix. The diff column already captures the full creation (every field "added" against an empty hashref), so no information is lost. Test plan: 1. prove t/db_dependent/Koha/Biblio.t 2. Observe that the "CataloguingLog MARC-in-JSON diff tests" subtest now also asserts: - ADD info == "biblio" - MODIFY info matches /^biblio \{/ - DELETE info matches /^biblio \{/ 3. Create / modify / delete a record via the staff UI with CataloguingLog on, then check action_logs: the ADD row's info column should read just "biblio". -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #14 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- (In reply to Janusz Kaczmarek from comment #8)

It seems to me that storing data in the Info field on ADD is inconsistent with the general principle that, for biblio, authorities, and items, this column contains the record state *before* modification. I think that, as before, in the case of ADD this content should be empty.

Hmm, yes, this was an unintentional side effect.. I've dropped it again in the followup.

On the other hand, adding Info on DELETE is valuable -- an ordinary librarian no longer needs to refer to deletedbiblio_metadata.

This one was intentional, and I've left it in :)

What do we think about continuing to store the data here, as before, using as_formatted, given that the overhead is small? Or perhaps we could transform the Info data on the fly, just for display, into a more human-readable format?

I need to keep the structure format for generating the diff so the easiest approach here was to improve the rendering in viewlog.. see follow-up for that :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #197923|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #197925|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #15 from David Nind <david@davidnind.com> --- Created attachment 198041 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198041&action=edit Bug 42188: Use MARC-in-JSON for biblio action log diffs Replace the custom _marc_record_to_diffable helper with MARC::Record::MiJ's to_mij_structure method, which produces standard MARC-in-JSON. This fixes three problems with the old approach: - Dollar signs used as subfield delimiters in the formatted string were ambiguous when subfield content itself contained " $x " sequences. - Grouping fields by tag into a hash lost inter-field ordering, making lossless reconstruction of the MARC record impossible. - Non-ASCII characters were subject to Perl string context encoding issues; JSON serialisation of the MiJ structure handles UTF-8 cleanly. _unblessed_for_log in Koha::Biblio is updated to return only the three biblio columns that are genuinely independent of the MARC record (frameworkcode, opac_suppressed, serial). The remaining columns (author, title, subtitle, notes, etc.) are denormalised copies of MARC data already captured in the _marc key, so including them produced redundant and noisy diffs. The fmt() helper in viewlog.js is updated to render JavaScript arrays as ordered lists rather than objects with numeric keys, improving the display of MARC subfield arrays in the diff table. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #17 from David Nind <david@davidnind.com> --- Created attachment 198043 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198043&action=edit Bug 42188: Use to_json for biblio action log info column Replace Dumper with to_json for the biblio info column in logaction, fixing UTF-8 encoding and extending coverage to ADD and DELETE actions which previously logged only the literal string "biblio". All three actions (ADD, MODIFY, DELETE) now log "biblio " followed by a pretty-printed JSON representation of the relevant state: pre-change for MODIFY, new state for ADD, and deleted state for DELETE. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #19 from David Nind <david@davidnind.com> --- Created attachment 198045 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198045&action=edit Bug 42188: (follow-up) Pretty-render biblio info column in log viewer The CATALOGUING info column for MODIFY/DELETE logs stores "biblio " followed by a pretty-printed JSON dump of the biblio hashref, which includes a MARC-in-JSON _marc structure. Raw, this is noisy and hard for librarians to read in the log viewer. Extract the nested-object formatter already used by the struct-diff renderer into a shared fmtJsonValue() helper, and use it from a new renderCatalogInfo() that runs over each div.loginfo on page load. If the cell text matches "biblio {JSON}" it is replaced with the pretty rendering; otherwise the cell is left untouched so non-catalog info payloads (and the bare "biblio" logged on ADD) are unaffected. Falling back silently on a parse failure means the old non-JSON Dumper-format item payloads continue to display as-is. Test plan: 1. Enable CataloguingLog. 2. Create, edit, and delete a biblio record via the staff UI. 3. Go to Tools -> Log viewer, filter by module=Cataloguing. 4. Verify: - ADD row's info cell shows just "biblio". - MODIFY and DELETE rows' info cells render as a nested list (leader, fields array of tag/indicator/subfield objects, plus frameworkcode/opac_suppressed/serial). - Diff column still renders the before/after struct-diff table. 5. Check another module's log entries (e.g. MEMBERS) are untouched. Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA QA Contact|testopia@bugs.koha-communit |lisette@bywatersolutions.co |y.org |m --- Comment #20 from Lisette Scheer <lisette@bywatersolutions.com> --- Works great and is now working as expected again. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198042|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198044|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #21 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 198054 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198054&action=edit Bug 42188: Use MARC-in-JSON for biblio action log diffs Replace the custom _marc_record_to_diffable helper with MARC::Record::MiJ's to_mij_structure method, which produces standard MARC-in-JSON. This fixes three problems with the old approach: - Dollar signs used as subfield delimiters in the formatted string were ambiguous when subfield content itself contained " $x " sequences. - Grouping fields by tag into a hash lost inter-field ordering, making lossless reconstruction of the MARC record impossible. - Non-ASCII characters were subject to Perl string context encoding issues; JSON serialisation of the MiJ structure handles UTF-8 cleanly. _unblessed_for_log in Koha::Biblio is updated to return only the three biblio columns that are genuinely independent of the MARC record (frameworkcode, opac_suppressed, serial). The remaining columns (author, title, subtitle, notes, etc.) are denormalised copies of MARC data already captured in the _marc key, so including them produced redundant and noisy diffs. The fmt() helper in viewlog.js is updated to render JavaScript arrays as ordered lists rather than objects with numeric keys, improving the display of MARC subfield arrays in the diff table. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #23 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 198056 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198056&action=edit Bug 42188: Use to_json for biblio action log info column Replace Dumper with to_json for the biblio info column in logaction, fixing UTF-8 encoding and extending coverage to ADD and DELETE actions which previously logged only the literal string "biblio". All three actions (ADD, MODIFY, DELETE) now log "biblio " followed by a pretty-printed JSON representation of the relevant state: pre-change for MODIFY, new state for ADD, and deleted state for DELETE. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 --- Comment #25 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 198058 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198058&action=edit Bug 42188: (follow-up) Pretty-render biblio info column in log viewer The CATALOGUING info column for MODIFY/DELETE logs stores "biblio " followed by a pretty-printed JSON dump of the biblio hashref, which includes a MARC-in-JSON _marc structure. Raw, this is noisy and hard for librarians to read in the log viewer. Extract the nested-object formatter already used by the struct-diff renderer into a shared fmtJsonValue() helper, and use it from a new renderCatalogInfo() that runs over each div.loginfo on page load. If the cell text matches "biblio {JSON}" it is replaced with the pretty rendering; otherwise the cell is left untouched so non-catalog info payloads (and the bare "biblio" logged on ADD) are unaffected. Falling back silently on a parse failure means the old non-JSON Dumper-format item payloads continue to display as-is. Test plan: 1. Enable CataloguingLog. 2. Create, edit, and delete a biblio record via the staff UI. 3. Go to Tools -> Log viewer, filter by module=Cataloguing. 4. Verify: - ADD row's info cell shows just "biblio". - MODIFY and DELETE rows' info cells render as a nested list (leader, fields array of tag/indicator/subfield objects, plus frameworkcode/opac_suppressed/serial). - Diff column still renders the before/after struct-diff table. 5. Check another module's log entries (e.g. MEMBERS) are untouched. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |26.05.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Keywords| |additional_work_needed -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Passed QA -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42188 Jacob O'Mara <jacob.omara@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Needs documenting --- Comment #30 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- This will not be backported to 25.11.x due to it being deemed an enhancement or a dependancy not being met -- You are receiving this mail because: You are watching all bug changes.