[Bug 27730] New: Innocent changes to framework silently delete bibliographic data
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Bug ID: 27730 Summary: Innocent changes to framework silently delete bibliographic data Change sponsored?: --- Product: Koha Version: 19.11 Hardware: All OS: All Status: NEW Severity: major Priority: P3 Component: Cataloging Assignee: koha-bugs@lists.koha-community.org Reporter: ian.bays@ptfs-europe.com QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl If you copy and change a framework and either delete a subfield in the new framework or mark the subfield tab as "ignore", then any bib data with data in that subfield, when saved in the new framework will be silently destroyed with no option to undo. Test Plan to confirm behaviour: 1. Create bib record in default framework with minimal data, but add tag 365 (as it has a number of subfields all visible) and populate subfields a through h with "fa" through "fh". Save the record and with sql check the contents of the metadata. (I found this SQL gives enough information substituting the correct bib number of the saved record: select tagfield, tagsubfield, frameworkcode, tab, hidden, (select ExtractValue(metadata, '//datafield[@tag="365"]/subfield[@code>="a"]') from biblio_metadata where biblionumber = 37002 and (select frameworkcode from biblio where biblionumber = 37002) = marc_subfield_structure.frameworkcode) tv from marc_subfield_structure where frameworkcode in ('', 'TEST') and tagfield = '365'; ) 2. Copy the default framework to a TEST framework and set (say 365$b to tab ignore and delete 365$d). (I used export the default framework to a csv, create new TEST framework and import from the saved default csv) 3. Edit the bib record and change framework to TEST framework (settings > change framework) and save the record. 4. Run the SQL again and observe the data from subfields b and d have both disappeared from the metadata. If you were to rerun these tests you may need to reload memcache to ensure the newly changed frameworks are picked up. I understand it has been broadly known by some long-term Koha users that deleting a subfield from the framework has this effect, but both these situations potentially cause bib data to be silently and permanently delated from the database. Many users will be unaware and may not discover the problem for some time. Potential solutions might include: a) For subfields not in the framework or with tab "ignore" allow the data to be saved as is on saving for edit; b) When selecting "change framework" detect if data would be lost and alert the user with an option to back out of the change without losing data; c) Do not allow deletion of a subfield from a framework and do not allow the "ignore" option for the tab. If a field is to be ignored but hidden there are plenty of visibility options in the framework. NB: The current delivered default framework has 13 subfields set to the "ignore" tab. I cannot see what purpose these serve as no data can survive within them. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Janet McGowan <janet.mcgowan@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |janet.mcgowan@ptfs-europe.c | |om -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Lucy Vaux-Harvey <lucy.vaux-harvey@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucy.vaux-harvey@ptfs-europ | |e.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|major |enhancement CC| |jonathan.druart@bugs.koha-c | |ommunity.org, | |katrin.fischer@bsz-bw.de Version|19.11 |master --- Comment #1 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- This is the historical behaviour, and how it is intended to work. It could be nice to have an alert (that could be switched off the a syspref?), but I am considering it an enhancement request. Katrin, do you agree? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Actually I was not aware of the setting "ignore" having this effect. The known thing is that you don't want to delete any field and subfield definitions as that will cause data loss. That's not great, but we have some documentation and warnings. For the "ignore" tab, I always assumed this was only a display setting as it's used to set the tab something is displayed on. So I think reading "Ignore" there as "don't display" is to be expected from user side. 1) Can we determin if the "ignore" always had this effect? I believe I have set "ignore" for some item fields in one of our oldest installations, but we never noticed any misbehaviour. 2) If we think this is not only to affect display, we definitely need to add at least a hint GUI wise. Maybe we should also add a list of "ignored" to the Bibliographic framework check to help people verify. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 --- Comment #3 from Ian Bays <ian.bays@ptfs-europe.com> --- Hi. The initial description includes a test plan that shows "ignore" deletes metadata. This was tested on recent versions including 20.11, but for completeness I went back to a 17.11 version and used a simpler method to verify the behaviour: We often set up a framework for (say) BK that sets (eg) the 022$a to ignore so I first checked this was the case on the system, then added a bib in the default framework and added an 022$a. I checked the biblio_metadata and it was present. I edited the record and changed frameworkcode from default to BK and saved. On inspection the 022$a has not present. Here is the SQL from this sequence: MariaDB [koha_ras]> select metadata from biblio_metadata where biblionumber = 37441; | <?xml version="1.0" encoding="UTF-8"?> <record xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" xmlns="http://www.loc.gov/MARC21/slim"> <leader>00242nam a22001217a 4500</leader> <datafield tag="999" ind1=" " ind2=" "> <subfield code="c">37441</subfield> <subfield code="d">37441</subfield> </datafield> <controlfield tag="003">OSt</controlfield> <controlfield tag="005">20210323092844.0</controlfield> <controlfield tag="008">210323b ||||| |||| 00| 0 eng d</controlfield> <datafield tag="022" ind1=" " ind2=" "> <subfield code="a">1234</subfield> </datafield> <datafield tag="100" ind1=" " ind2=" "> <subfield code="a">Test</subfield> </datafield> <datafield tag="245" ind1=" " ind2=" "> <subfield code="a">Test</subfield> </datafield> <datafield tag="942" ind1=" " ind2=" "> <subfield code="2">ddc</subfield> <subfield code="c">MONO</subfield> </datafield> </record> | MariaDB [koha_ras]> select frameworkcode from biblio where biblionumber = 37441; +---------------+ | frameworkcode | +---------------+ | | +---------------+ MariaDB [koha_ras]> select tab from marc_subfield_structure where frameworkcode = '' and tagfield='022' and tagsubfield = 'a'; +------+ | tab | +------+ | 0 | +------+ MariaDB [koha_ras]> select frameworkcode from biblio where biblionumber = 37441; +---------------+ | frameworkcode | +---------------+ | BK | +---------------+ MariaDB [koha_ras]> select tab from marc_subfield_structure where frameworkcode = 'BK' and tagfield='022' and tagsubfield = 'a'; +------+ | tab | +------+ | -1 | +------+ MariaDB [koha_ras]> select metadata from biblio_metadata where biblionumber = 37441; | <?xml version="1.0" encoding="UTF-8"?> <record xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" xmlns="http://www.loc.gov/MARC21/slim"> <leader>00216nam a22001097a 4500</leader> <datafield tag="999" ind1=" " ind2=" "> <subfield code="c">37441</subfield> <subfield code="d">37441</subfield> </datafield> <controlfield tag="003">OSt</controlfield> <controlfield tag="005">20210323094228.0</controlfield> <controlfield tag="008">210323b ||||| |||| 00| 0 eng d</controlfield> <datafield tag="100" ind1=" " ind2=" "> <subfield code="a">Test</subfield> </datafield> <datafield tag="245" ind1=" " ind2=" "> <subfield code="a">Test</subfield> </datafield> <datafield tag="942" ind1=" " ind2=" "> <subfield code="c">MONO</subfield> </datafield> </record> | 1 row in set (0.00 sec) MariaDB [koha_ras]> select value from systempreferences where variable like '%version%'; +------------+ | value | +------------+ | 17.1109000 | +------------+ -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Thx Ian. What do you think we shoudl do? Make this a display thing (only affecting the tab) or a warning/hint be ok? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 --- Comment #5 from Ian Bays <ian.bays@ptfs-europe.com> --- As yourself I had assumed that "ignore" would leave the data intact, so I initially thought it would be "best" if the data is preserved but it is just not pulled out for display. Because we understand it may take time to agree and implement we set about scripting changes to our frameworks to set all those that say "ignore" to a real tab but to set "hidden" to a value that hides it. However we discovered that the default framework has circa 13 subfields set to ignore including 999$a, $b and $c. If you set these to other than ignore the framework check fails. We have an amended script that reinstates the real tab number for all set to "ignore" except the 999 subfields. The script also copies tags and subfields from default where they had been deleted (or possibly not updated from the delivered defaults). Many people will never notice losing these bits of data as they don't see them, but if you've got your data from a rich source and save in a different framework you would not expect the rich data to be removed. A warning (if you are changing to a framework that will lose data for this record) might be a good first step until there is consensus on how to deal with it. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- With Bug 31923 - 'Ignore' tab description is misleading there is no at least a warning in the framework configuration. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=31923 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Lauren Denny <lauren_denny@sil.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lauren_denny@sil.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |a.roussos@dataly.gr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |esther.melander@bywatersolu | |tions.com --- Comment #7 from Phil Ringnalda <phil@chetcolibrary.org> --- *** Bug 35135 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Michelle Spinney <mspinney@clamsnet.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mspinney@clamsnet.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Heather <flyingendpaper@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |flyingendpaper@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 --- Comment #8 from Lauren Denny <lauren_denny@sil.org> --- It seems like not all subfields set to 'ignore' are deleted. The 952 $6 & $9, 999$a, $b, $c, $d need to be set to ignore or the MARC Framework test will report errors. It feels like Koha is using these fields for system data and the 'ignore' tab allows it to do this. It would be helpful if the frameworks mentioned that the fields the system uses should not be touched or a new option for 'system' instead of 'ignore be used. Currently the tab 'ignore' can mean two things. One means it will delete the data from the record if the framework has the subfield set to ignore. The other is something that the system uses and no one should touch. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27730 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=39560 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org