https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 --- Comment #1 from Bernard <bernard.scaife@openfifth.co.uk> --- Created attachment 183742 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183742&action=edit Bug 40300: Add chop punctuation to PartName () in Part xslt templates -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 Alexander Wagner <alexander.wagner@desy.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alexander.wagner@desy.de --- Comment #3 from Alexander Wagner <alexander.wagner@desy.de> --- Probably a bit philosophical: While I am personally all for removing punctuation from records and adding them automatically, I do wonder if in your case Koha is not entering a strange mixture. IMHO - if a punctuation _is there_ the logic should be to keep and use it. - if no punctuation is there it should add it. IOW while I see what your code does and I also perfectly agree that having ` ; ; ` is wrong _and_ ugly, I do wonder why the ` ; ` in `|p` is different from the `.` in `|a` or `|v` and thus treated differently. So, I think instead of chopping the punctuation from `|p` and add it again automatically, the code should look at leader 18 and decide if it should do anything wrt the punctuation in the first place. And if the leader says "all chars are there" it should just do nothing and leave it as is. (My guess would be your records states that it holds those chars.) BTW: I started working on automatic punctuation in https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21303. After playing around with XSLT and some discussion with Cait we came to the conclusion (which might be wrong) that it is probably _a lot_ easier to do this punctuation stuff by means of a Marc Filter in Perl especially as there are some complex things out there. Cf https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37196, https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37325 and their tests for examples and an implementation for a few fields. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|oleonard@myacpl.org |bernard.scaife@openfifth.co | |.uk --- Comment #5 from David Nind <david@davidnind.com> --- Also added you as the assignee 8-) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 Bernard <bernard.scaife@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #183742|0 |1 is obsolete| | --- Comment #6 from Bernard <bernard.scaife@openfifth.co.uk> --- Created attachment 183830 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183830&action=edit Bug 40300: Add chop punctuation to PartName () in Part xslt templates In both intranet and OPAC full detail display an 830 series field which has a part $p displays multiple semi-colons if the field contains one. There is a part template which needs to use chopPunctuation. Test plan: 1. Add series field 830 with $a Analecta Gregoriana, Series Facultatis Philosophiae. $p Sectio B ; $v n. 14. 2. Observe OPAC or intranet detail record: Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; ; n. 14. 3. Apply patch and restart_all 4. Observe OPAC or intranet detail record again - should be Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; n. 14. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|25.05 |Main Text to go in the| |This fixes the display of release notes| |the 830 series tag in the | |staff interface and OPAC. | |An 830 series tag which has | |a part $p subfield displays | |multiple semicolons if the | |subfield already contains | |one. | | | |Example: | |- A record with an 830 tag | |with these subfields: | | $a Analecta Gregoriana, | |Series Facultatis | |Philosophiae. | | $p Sectio B ; | | $v n. 14. | |- Before this patch is | |shows as: Analecta | |Gregoriana, Series | |Facultatis Philosophiae. | |Sectio B ; ; n. 14. | |- After the patch is shows | |as: Analecta Gregoriana, | |Series Facultatis | |Philosophiae. Sectio B ; n. | |14. --- Comment #9 from David Nind <david@davidnind.com> --- I've signed off. However, I'm not knowledgeable enough about cataloging conventions and Koha to know what the "correct" way to do this is. I know it has been discussed before, but the nuances are over my head 8-). It may pay to ask in the Koha Mattermost cataloging channel to get some feedback from catalogers. Testing notes (using KTD): 1. Edit the BKS framework to make the 830 $a, $p, and $v subfields visible in the editor. 2. Edit a record in the sample data that uses the BKS framework, for example "Programming Perl". -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 --- Comment #11 from Alexander Wagner <alexander.wagner@desy.de> --- (In reply to Bernard from comment #7)

(In reply to Alexander Wagner from comment #3)

...

Probably a bit philosophical: While I am personally all for removing punctuation from records and adding them automatically, I do wonder if in your case Koha is not entering a strange mixture. IMHO

- if a punctuation _is there_ the logic should be to keep and use it. - if no punctuation is there it should add it.

[...]

Interesting. I'm all for anything that can make this simpler and I like your idea of a separate punctuation.pm. I would be reluctant to have to rely on the LDR position 18 value as experience shows this may be set to "a" but that not be the reality.

Hm, but we agree that the records you have in mind are "broken" wrt the idea of Marc in this regard. Shouldn't one fix the broken record then (IOW fix the leader)? I think in general one will have a mixture of records in any given system over time, those that hold the punctuation and those that don't, right? They may just flood in during cataloguing if you ingest records from various sources. (AFAIK the SRU/Z39.50 importers do _not_ strip or add punctuation, so it depends on the source what you get.)

Perhaps a syspref making it an option?

Uh. This adds another unrelated point to configure what to do with those funny chars and it would apply globally as well. One could head for a switch for - follow the leader - ignore the leader and treat as "records have punctuation" - ignore the leader and treat as "records don't have punctuation" Where in the last case you'd chop everything "just in case". This sounds a bit weird.

I guess the issue is if we currently already use chopPunctuation

I noticed that, indeed. I was wondering all the time if this is not fixing a fix for a fix that might get another layer of plaster as soon as another issue pops up ;) That's the sole reason why I commented on this bug here that I noticed quite a few occurrences of chopping while some fields never see it.

at all in the xslt (we do), it should do it everywhere required.

When I looked at Koha sample records for the test cases to `punctuation.pm` I found, that "strange things happen in cataloguing" that will not allow to add punctuation in _all_ corner cases automatically. Or actually the other way round: it is well possible and following the rules to construct a record that will not allow automatic punctuation. Hence, for those cases it might be necessary to add the punctuation on record level. But to give a proper display in those cases you'd have to leave the punctuation alone. I think this is the reason behind the leader-switch.

Or else not at all.

I tend to believe "not at all" would be the way to go. Or at least if it's done in xslt it should follow the leader and if the display is wrong the offending leader can be fixed easily.

For now, pragmatically, I think I'd rather stick to fixing what we have.

This is also ok with me if Koha thinks this is the way to go. I just see some more issues down the winding road as chopping assumes you can _always_ add the punctuation automatically. (If so, why not strip it entirely on ingest and set the leader properly ;) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 --- Comment #14 from Alexander Wagner <alexander.wagner@desy.de> --- (In reply to cgresser from comment #13)

At the moment Koha seems to fix globally punctuation "problems" which are not a problem for a cataloguer who does use Marc21 rules! I had previously [...] If the record is AACR2 as opposed to RDA, then I believe that every 300 field should end in a full-stop. Always!

I think this is one of the examples where you could not (easily) do the punctuation automatically in general. It's a bit along my argument: I perfectly agree that IT people would have written cataloguing rules differently, but as you say, they are as they are.

[the full-stop removal is less of an issue, as most people looking at our records do not know what should be there, but it is a point in case for Koha globally changing the display of records it doesn't understand the full syntax of!)

IMHO _in general_ one _can't_. There are too many exceptions. Sometimes, it's hard for me to get cataloguing and rules into one sentence. In some areas it sounds more like a list of exceptions. Like German grammar ;)

I love KOHA a lot, but for Koha globally to chop punctuation off for end of [...] adding things (the semi-colon), where we human beings do a good job of following syntax. This doesn't make sense.

I'd see a few more nuances here. In Marc there _is_ actually a mechanism to tell if you should add punctuation or not on a per record level. This is said leader (cf. https://www.loc.gov/marc/bibliographic/bdleader.html): 18 - Descriptive cataloging form # - Non-ISBD a - AACR 2 c - ISBD punctuation omitted i - ISBD punctuation included n - Non-ISBD punctuation omitted u - Unknown I would argue, that if you set the leader 18 to `c` or `n` explicitly, then you as a cataloguer say: "I don't bother about punctuation. Dear computer do it for me as best you can." And it's your cataloguers conscious decision. So, IMHO in these cases Koha should try as best as it can to add punctuation. OTOH, it's better to leave the stuff _as is_ as our cataloguer told the system "don't touch the punctuation, I already did it". And if the display is still wrong IMHO it's time to _fix the record_. (By setting the leader appropriately and/or fixing the punctuation.)

If some Marc data has "mistakes" in them, either a diligent human being needs to deal with them individually, or if desired with a global change for _their_ catalogue/data.

The Marc-approach seems to me even better than a global default as it is on the _per record_ level. E.g. one may ingest data from various sources via Z39.50, SRU, harvesting etc. So it may well happen that the source systems approach to punctuation differs. I learned that the union catalogue we use as our main source adds punctuation automatically so "non in the record" all the time, while my impression is that LoC _adds_ the punctuation to the records "all the time". Now we ingest from LoC if our union catalogue can't deliver => in general I end up with per record rules wrt punctuation. Without obeying the leader my fellow colleagues have to check punctuation in every record and fix it in every other or so. So, I think it would be nice to follow the data and act accordingly.

Applying a fix for some data globally, for everyone who uses Koha uses a dictatorial approach which I haven't thought was Koha's intention.

That's a bit harsh. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 cgresser <cgresser@jesuit.org.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |In Discussion -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 --- Comment #17 from Alexander Wagner <alexander.wagner@desy.de> --- (In reply to Bernard from comment #16)

As I am not a cataloguer, my understanding of some of the detail here is no doubt hazy. However, I think in a nutshell we are saying we don't want to break well catalogued records but we'd like Koha to try to fix ones that are not, or at least to some degree.

I think this is the gist of it. BTW: just noticed that in #21303 Caroline stumbled upon the automatic fixing via XSLT. (I knew I already read a bug on this, that's was reason for my initial comment.)

Given that the rules may vary by field, it seems to me that using a separate punc.pm as drafted in bugs #21313, #37196 and #37325 is going to be the best way to achieve this and not make the xslt unwieldy.

Indeed, my first approach was by XSLT, but I was happy to learn from Cait that Mark filters exist and could be used for this, even though right now the filters applied are in a static array in the code. (Still the use case would be general enough to justify a change to the list, I think)

Not that this is a trivial piece of work of course.

Agree. I split it up by fields as I think it would allow for a pragmatic approach adding fields as we stumble upon them. Probably fixing up the XSLT on the way. So Koha could improve over time without blocking everything by a huge overhaul. But maybe there is a better approach?

I would favour an approach where the each relevant field is presented to punc.pm. If it is "correct" anyway, nothing would be changed and if it is not, then it would be fixed up and returned. This way we do not need to consider leader 18 at all.

I do not yet get how you know if a record is "correct"? Say, I take a tag and pass it though the automatic punctuation and compare it to the punctuation presented. It could be the same or differ. But what is "correct"? If we treat "the same" as correct, we are implicitly enforcing the rules of `Punctuation.pm` to all records as all those that differ get rewritten and those that are the same already follow the very rules. I could imagine a system setting, if you want a way to disregard the leader. (Seems you have good reason to distrust it.) Something along the lines: RewritePunctuation: - Never -> do nothing - Always -> chop and add - Evaluate leader 18 -> respect the leaders value on a per record basis Looking at your initial issue one could ponder if a general rule makes sense that checks if the same char appears twice. E.g. if the punctuation char is ` ;` one could replace ` ; ;` by ` ;`. But I am not sure if there isn't a strange case, I am not aware of, where something like this could even be "correct". -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40300 --- Comment #19 from cgresser <cgresser@jesuit.org.uk> --- Thank you, Bernard. Another way to look at this, whether you are working with computer code for Koha, etc. or with the conventions and rules for cataloguing a book, journal, etc. on Koha. Imagine you are doing everything right. And then Koha comes along and adds or removes punctuation, even though what you have entered in Koha cataloguing is correct. To me it still feels like humans have told Koha to overrule other humans' work, based on an assumption that Koha knows best, when Koha has been misinformed by humans about these issues of punctuation. Basically, Koha is ignorant of the differences of how people catalogue (a significant number of people in the English-speaking world use Marc21), but dictates one approach to all people using Koha. Surely that must be seen as wrong, and against the spirit of Koha offering diversity and options to configure things. Koha's behaviour here is not open, it is set on satisfying the needs and helping one point of view of how to catalogue. Can this please be considered in principle by someone who develops Koha, and makes decisions what Koha is supposed to do and not? -- You are receiving this mail because: You are watching all bug changes.