https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 Alejandra Tenorio <a.tenorio@cgiar.org> changed: What |Removed |Added ---------------------------------------------------------------------------- URL| |https://adm-knowledgecenter | |.cimmyt.org/cgi-bin/koha/ca | |talogue/detail.pl?biblionum | |ber=67744 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- You say it loses data after being edited. Did you alter your MARC frameworks? What data is lost? Are the fields/subfields lost being present in the framework? Are they set to be "ignored" or have been deleted? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #3 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- If only some of many are lost, it's probably not the frameworks. It might be some kind of length limit at play here. Like maybe the number of fields to display? Adding David as he has a good idea about the inner workings of our MARC conversions I think. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Data loss when editing MARC |Duplicate "id" and "name" |records |inputs in basic editor | |leads to data loss --- Comment #5 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #4)

Nothing comes to mind off the top of my head.

Sounds like this needs more troubleshooting/investigation.

Usually I'd point you at https://koha-community.org/support/ for this one, but I decided to take a little look anyway. -- I looked at the MARCXML file and it has 1685 700 fields. Imported... still has 1685 fields. Trying to edit... and the basic editor doesn't load properly. I can see in the developer tools in the browser that 26 fields have duplicate "id" and "name" attributes. So that would explain your data loss. I don't know why a person would even try to edit this record. It's fairly clear at a glance that the editor isn't loading properly. -- While 1600-2000 authors is a bit of an edge case for many Koha instances, it's not unheard of in research publications. It would be interesting to find the threshold where duplicate ID/name fields occur. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Duplicate "id" and "name" |Duplicate "id" and "name" |inputs in basic editor |inputs in MARC basic editor |leads to data loss |leads to data loss -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |phil@chetcolibrary.org --- Comment #8 from Phil Ringnalda <phil@chetcolibrary.org> --- (In reply to David Cook from comment #5)

It would be interesting to find the threshold where duplicate ID/name fields occur.

That's a statistics question: how often does /cataloguing/addbiblio.pl have to call int(rand(1000000) to get the same number twice? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to Alexander Wagner from comment #7)

Quite common in experimental particle physics ie. every Atlas, CMS, Alice paper these days. (Atlas is currently at ~2.9k, CMS at ~2.3k)

Just to add, that in join² we recently discovered some papers with ~35k "authors". They are not from HEP but from medicine, usually associated with COVID research. (These do cause a lot of headache at a number of other services downstream as well.)

I suppose it depends on the type of library and how they use Koha. For instance, a circulating public library is probably not going to run into this. An academic library will probably have them in a digital repository and served via a discovery system. A special library hopefully has a digital repository but might be using their Koha for it. The smaller special libraries not using digital repositories are the ones where I'd usually see this scenario. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to Alejandra Tenorio from comment #9)

Trying to edit... and the basic editor doesn't load properly. You can edit if the page is zoomed to 70%.

Thanks for the tip.

--------- I don't know why a person would even try to edit this record. It's fairly clear at a glance that the editor isn't loading properly. Because we have to associate each author with to their authority.

Sorry I meant that needing to zoom the page to 70% might be an indication that Koha isn't expecting to work this way, so it might not be surprising if it didn't work correctly. Just a miscommunication on my part. No worries. -- The MARC editor rarely gets attention, so I think a lot of people will want to avoid this issue. I don't mind looking at it, but I probably won't have time for a little while. Even when someone does provide a patch, it might take some time to get it approved. So workarounds to help you for now... you could try the Advanced Editor but I don't know if it'll have similar problems. (Phil, you might have some insight here?) Alternatively, you can do your cataloguing edits in an external problem like MarcEdit and upload (using a 999$c record matching rule to replace the existing record). Hopefully someone else can fix this before I can, but that's probably the best help I can provide right now. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- (In reply to Phil Ringnalda from comment #13)

I was looking at whether the advanced editor would handle it better, but I got stuck on the fact that the basic editor is apparently not actually losing data, it's just mangling it. When it winds up with duplicate IDs, it then saves two or three 700 fields together in just one. That's invalid MARC, you can't have duplicate $a subfields, but the basic editor doesn't care if an existing record is invalid, it will still let you resave it. The advanced editor does care, so it's not possible to test how it behaves without someone unmangling all the fields that currently have duplicate subfields first.

If you don't laugh you cry... haha. Thanks, Phil. I can take a look at that tomorrow. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #16 from Alexander Wagner <alexander.wagner@desy.de> ---

...

Just to add, that in join² we recently discovered some papers with ~35k "authors". [...] I suppose it depends on the type of library and how they use Koha.

Of course. I just added this comment to give an idea how far one would need to scale up the number of authors to be done with (for a while). Besides getting the ids right there might even be some issues rendering the display of whatever editor you use, btw.

For instance, a circulating public library is probably not going to run into this.

Of course not. [...]

A special library hopefully has a digital repository but might be using their Koha for it.

We were pondering on using Koha as the successor system for our current repository, that is also based on Marc21 and we have this several k author lists use a lot of normalizations etc. Koha would fit a lot of bills. It would however require Koha to get a decent OA repository component. (join² was pondering if we could pull this off, but it was finally decided otherwise for other reasons.) Still, I think it would be worthwhile to add a decent OA repository component to Koha, but that's another story and also quite a task to get it right. It is actually quite convenient if your repo is your catalogue. I'll miss that.

The smaller special libraries not using digital repositories are the ones where I'd usually see this scenario.

It's probably not a matter of size of the library, but on the research area of your institution. I would definitely call us pretty small, while Atlas and CMS papers are regulars in our systems. (More on the repository end / article cataloguing of course.) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- (In reply to Alexander Wagner from comment #16)

I just added this comment to give an idea how far one would need to scale up the number of authors to be done with (for a while). Besides getting the ids right there might even be some issues rendering the display of whatever editor you use, btw.

Mmm I understand you now. Could also cause issues rendering on the detail page and search results. Could be a good reason to build in some "length handling" for some of the interfaces.

We were pondering on using Koha as the successor system for our current repository, that is also based on Marc21 and we have this several k author lists use a lot of normalizations etc. Koha would fit a lot of bills. It would however require Koha to get a decent OA repository component. (join² was pondering if we could pull this off, but it was finally decided otherwise for other reasons.)

Still, I think it would be worthwhile to add a decent OA repository component to Koha, but that's another story and also quite a task to get it right. It is actually quite convenient if your repo is your catalogue. I'll miss that.

I hope some day that Koha is more modular so that it would be easier to add in these components. Although part of the problem is maintenance. It's not easy for a small number of volunteers to maintain such a large and growing codebase. Tough to be all things to all people... But I'd be interested to hear more.

It's probably not a matter of size of the library, but on the research area of your institution. I would definitely call us pretty small, while Atlas and CMS papers are regulars in our systems. (More on the repository end / article cataloguing of course.)

What I mean is larger institutions usually have more money which means they can pay for more systems. But that's a good point. I do support some library systems for small low budget parts of large wealthy organisations. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #20 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 170764 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170764&action=edit Bug 37675: Make random generates the same numbers less probable This is really not the ideal patch but definitely the easiest to write and with (hopefully) the least side-effects possible. Ideally we would like to rethink how this "id" is generated and start from ... 1 To improve this patch we could follow-up and store the list of the generate numbers, then generate a new one if it exists already. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|23.11 |unspecified Status|NEW |In Discussion -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #23 from David Cook <dcook@prosentient.com.au> --- (In reply to Phil Ringnalda from comment #22)

Interesting thing about that patch: it doesn't help, nor does adding two more zeros, nor does adding five more zeros.

In hindsight, maybe I re-named this bug prematurely. I assumed it was caused by the duplicate "name" inputs, but I didn't actually verify it... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #25 from Phil Ringnalda <phil@chetcolibrary.org> --- Don't sell the duplicate ids short, just because they aren't caused by the size of the rand() call. I just loaded the record with duplicate ids for Alalalmeh, S.O. and Pashaei, A., saved the record, reloaded it, and now I still have a field for Alalalmeh, S.O. and no field for Pashaei, A., which is exactly the reported problem. It's a shame that we can't get rid of the duplicate ids by throwing zeroes at rand(), but it doesn't mean they aren't the problem. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #27 from David Cook <dcook@prosentient.com.au> --- (In reply to Phil Ringnalda from comment #25)

It's a shame that we can't get rid of the duplicate ids by throwing zeroes at rand(), but it doesn't mean they aren't the problem.

I just updated cataloguing/addbiblio.pl to guarantee that CreateKey() would always return a unique value, and there are still duplicates. The interesting thing is that the duplicates always seem to be around the same number... always about 20-some fields with errors suggesting 10ish duplicate numbers. Seems like there's another factor here... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #29 from David Cook <dcook@prosentient.com.au> --- Created attachment 170795 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170795&action=edit Bug 37675: Ensure unique random number server-side This change ensures that a unique number is used for the random numbers used in the MARC basic editor. Note that there is a chance for a near infinite loop for records with a large number of records. This could be improved by dynamically increasing the random number max and throwing a fatal error if too much time is spent generating numbers. Or we could calculate how many numbers we need ahead of time, and use those. Note that the Javascript used for creating random numbers does not have knowledge of the already used set of numbers currently. This should be improved as well. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #31 from David Cook <dcook@prosentient.com.au> --- Created attachment 170796 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170796&action=edit Bug 37675: Remove need for random numbers in MARC subfields This change removes the need for random numbers in the index_tag and index_subfield variables. Instead, respectively, we use the tag number and an incrementing integer. This eliminates the possibility of duplicate ID/name values in server-side generated content. Note that there is still a risk of a collision when generating these values via the Javascript at this time. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #33 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to David Cook from comment #30)

I think Jonathan's patch didn't have much impact, because it didn't target the most necessary "create_key" function.

Good catch, I missed that one.

But it has its own problems as well. I don't like while() loops in web apps.

Yes, that's terrible. And the fact the a collision could still happen between server-side and client-side seems pretty bad as well. What about using UUIDs instead? :) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #35 from Jonathan Druart <jonathan.druart@gmail.com> --- Generating UUIDs is obviously slower but at least we don't create mess in different places. Patch for discussion, the editor will need to be tested carefully. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37675 --- Comment #37 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #35)

Generating UUIDs is obviously slower but at least we don't create mess in different places.

Patch for discussion, the editor will need to be tested carefully.

I tried adding an alphabetic character to the start of the server-side generated integer, but I started getting some weird Javascript errors. My patch worked, but stopped and the error messages went away. Haven't tried with the UUIDs yet, but I'd look for that when testing. (I need to do some customer work now...) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.