https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 Jonathan Field <jonathan.field@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|21.11 |master -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #2 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Nice one. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #4 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- I wondered if it was as simple as just running it through RecordProcessor.. but when I started to have a quick look at the code I couldn't wrap my head around it.. perhaps not as simple as it appears on the surface. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #3)

I think that we need a database column to store the 942$n data to make it easy to query via SQL.

I still think that this is the solution for this particular issue.

OpacHiddenItems will still be an issue but that can be a separate issue...

I actually just remembered that I actually have a solution for this! Check out the patch on bug 19704. I use SQL::Abstract to generate SQL based on OpacHiddenItems! (Note that the 942$n issue on that report is managed using ExtractValue() but it's not as efficient as just having a "biblio" or "biblioitems" column of "suppressed".) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Briefly looking at this again in the context of OpacHiddenItems... We'd need to either JOIN the items table to biblio_metadata OR we'd need to use a subquery. If the subquery were the same as the one used in the UNION, I wonder if it would just cache the 2nd run of the same query. That would be interesting to benchmark... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ere.maijala@helsinki.fi -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- Ah... but the $order_limit specifies a LIMIT which would be a problem if we were filtering out records using RecordProcessor... That said, we could move the record fetch into a function and run it again if the $count hadn't hit the $max. That would make the response even slower than it already would be using RecordProcessor though... But I can't think of a simple way to support "OpacHiddenItemsHidesRecord: Hide" except by post-processing outside the SQL. -- Another thought I had was that maybe we do need to store "hidden_in_opac" in the "biblio" and "items" DB table records. If someone updates "OpacHiddenItems", then they need to do a "touch all items". In theory, this is a system preference that shouldn't be frequently changed anyway. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|MARC Bibliographic data |Web services |support | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- I've opened bug 31725 to address the hiding of items using OpacHiddenItems. And it's got me thinking that the "deleted" status might be the way to go for records hidden using OpacSuppression or OpacHiddenItems. It's not a 100% perfect solution, but I think it could be a "good enough" solution. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- (In reply to Ere Maijala from comment #13)

I think it's the only solution that would work without forementioned side-effects. It's important to report the records as deleted particularly since they might not have been suppressed before. And that'd be the case for practically every vanilla Koha instance harvested with OAI-PMH up until this change.

I'm glad to hear that you agree!

That said, there are use cases where OAI-PMH is used for non-opac functionality and suppressed records should be included just like before.

Do you have some use cases in mind?

For flexibility it should be possible to configure suppression per OAI-PMH set in addition to a default setting.

GetRecord isn't affected by sets, so are you thinking of a flag in the "oai_sets" data that would disable the suppression/hiding of records? -- I'm planning to trial this hiding locally by using suppression/hiding by default, and then using a system preference to disable it (to cater for any use cases that come up where libraries might need the unfiltered records still). -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- All right... that wasn't so bad in the end. A bit of code in C4::Biblio::GetMarcBiblio, C4::Biblio::EmbedItemsInMarcBiblio, Koha::OAI::Server::GetRecord->new, and Koha::OAI::Server::Repository->get_biblio_marcxml -- There are no new database calls (only syspref calls to cache), so the only overhead added is CPU time and that will be negligible. -- One could argue that there are alternatives to my hashref parameter idea, but it works. I'll likely be implementing this locally in 21.11 tomorrow night. -- I've run out of time today but it shouldn't be a big drama to add unit tests (at least for the C4::Biblio code). -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #18 from Ere Maijala <ere.maijala@helsinki.fi> --- (In reply to David Cook from comment #14)

(In reply to Ere Maijala from comment #13)

...

I think it's the only solution that would work without forementioned side-effects. It's important to report the records as deleted particularly since they might not have been suppressed before. And that'd be the case for practically every vanilla Koha instance harvested with OAI-PMH up until this change.

I'm glad to hear that you agree!

...

That said, there are use cases where OAI-PMH is used for non-opac functionality and suppressed records should be included just like before.

Do you have some use cases in mind?

Anything non-OPAC could qualify, e.g. a union catalog based on OAI-PMH. Or even an OPAC instance for staff use only.

...

For flexibility it should be possible to configure suppression per OAI-PMH set in addition to a default setting.

GetRecord isn't affected by sets, so are you thinking of a flag in the "oai_sets" data that would disable the suppression/hiding of records?

GetRecord response has header/setSpec to indicate which sets the record belongs to. It should reflect suppression by excluding a set where the record would be marked deleted. So if we have sets "all" and "opac", a suppressed record would only have <header><setSpec>all</setSpec></header>. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marrone.patricio@gmail.com --- Comment #20 from David Cook <dcook@prosentient.com.au> --- *** Bug 19064 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #22 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #21)

Can we add 'supressed' and 'supressed_date' to the biblio table and make our high level methods take care of syncing things?

Can you elaborate on that? OpacSuppression could work with that, but I don't know how you'd make it work with OpacHiddenItems with OpacHiddenItemsHidesRecord. (Unless you require people to touch_all_biblios.pm after changing OpacHiddenItems and OpacHiddenItemsHidesRecord.)

That would make it easy to build a performant query for identifying (to be marked as) deleted records in OAI.

That could be interesting. It would need to be done via a join or a subquery, but it would still probably be fairly performant. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #24 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #23)

Just realized that while my changes work well for GetRecord and ListRecords, they don't for ListIdentifiers.

For that, we'll have to fetch the metadata in order to know whether or not it needs to be suppressed/hidden.

That is unless we do the analysis at the SQL level.

Ugh, and even fetching the metadata doesn't work for OpacHiddenItems unless that metadata format includes items. This is so frustrating. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #26 from David Cook <dcook@prosentient.com.au> --- For now, I will proceed with my patches to solve some immediate local problems, but they're probably not worth upstreaming as they don't comprehensively solve the problem. For that, I think we need to change how we handle OpacSuppression and OpacHiddenItems and to use SQL for the suppression/hidden lookup. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #28 from David Cook <dcook@prosentient.com.au> --- (In reply to Ere Maijala from comment #27)

Now to more practical stuff. The multiple endpoints idea works for me. I don't think there's any other interoperable way if you need to have different rules. You could also have several endpoints like 'internal/oai.pl', 'opac/oai.pl' and 'test/oai.pl', no problem. From the protocol point of view they'd just be separate providers. However, they should also have separate OAI identifiers. Things could go really bad if a harvester receives the same record from multiple endpoints.

According to the OAI spec, OAI identifiers are only unique within their repository/0provider, but I don't think a lot of OAI harvesters adhere to the OAI spec in that regard... so that's a good point. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #30 from David Cook <dcook@prosentient.com.au> --- (In reply to Ere Maijala from comment #29)

I'll soften my stance a bit. I suppose it boils down to whether two endpoints are considered the same repository. It might also be enough to just document that one shouldn't harvest from multiple endpoints like they were the same repository.

Yeah, I think documenting could be enough. We could even have separate system preferences. On the non-Koha system with private/public endpoints, the private one is restricted by additional security and used for internal projects. The public one is just open to the public, so very different consumers. But we also set that up with those separate endpoints from the initial launch. With Koha, it'll be harder to transition for existing consumers of the OAI-PMH... That said, people looking for support for suppressed/hidden records, they'll probably be doing it intentionally and thus be able to coordinate a migration to a new public endpoint more easily... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #32 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Any chance of sharing those patches David? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #34 from David Cook <dcook@prosentient.com.au> --- Looks like Koha has changed quite a bit since I wrote those patches so I'll need to refactor them anyway... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #35 from David Cook <dcook@prosentient.com.au> --- Hoping to look at this again in August. I took a look at my local code in prod, and it doesn't require a ton of code changes. Mostly just a case of sitting down, porting the code, and testing it works. Writing tests of course. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #36 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 170280 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170280&action=edit Bug 31161: Suppress OAI records when all items hidden by OpacHiddenItems This patch set adds an option to Koha::Metadata::Record to suppress a record if all items are hidden. This option is only enabled by OAI and only when OpacHiddenItemsHidesRecord is enabled. This test plan assumes KTD default data To setup: edit/create /etc/koha/sites/kohadev/oai.conf with content below: format: marcxml: metadataPrefix: marcxml metadataNamespace: http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim schema: http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd include_items: 1 Set sysprefs: OAI-PMH: Enable OAI-PMH:ConfFile: /etc/koha/sites/kohadev/oai.conf OAI-PMH:AutoUpdateSets: Enable OAI-PMH:AutoUpdateSetsEmbedItemData: Enabled OpacHiddenItems: itype: [BK] OpacHiddenItemsHidesRecord: Hide Adminsitration->OAI set configuration->New set Choose anything for setSpec and setName On new set: Actrion->define mappings Field: 942 Subfield: c Operator: is equal to Value: MU To test: 1 - Set above, do not apply patch, restart_all 2 - Visit: http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml 3 - Confirm record 1 is visible wit no items (or anothe record that has BK itemtypes only) 4 - perl misc/migration_tools/build_oai_sets.pl -r -v -i 5 - Confirm 11 records in set created above (or correct number for your site) 6 - Apply patches, restart_all 7 - Repeat 2, note "Results fetched" is 43-50, count is off because of hidden records 8 - Confirm record 1 and many others not shown 9 - Confirm 'Show more' works 10 - Repeat 4, still 11 11 - Find a record with record type music, say 315 12 - Edit all items and change type to BK - Books 13 - Repeat 4 - now 10 records 14 - Set pref OpacHiddenItemsHidesRecord: Don't hide 15 - Repeat 2 and 4 and confirm results as before patch -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #38 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens (kidclamp) from comment #37)

We will clearly need tests here, however, the fact that we now respect OpacHiddenItems makes this a bit more urgent - in the past outside systems could use the filtering to determine if a record should be shown, essentially implementing their own OpacHiddenItems - but now the items don't come through and we cannot externally determine if a record truly has no items, or if all items have been hidden.

Marking needs signoff as I need feedback on the approach and assumptions about following 'OpacHiddenItemHidesRecord'

I think that I might like my approach better. If I understand this patch correctly, if a record gets harvested by a consumer, then is hidden in Koha, the record will just disappear from Koha's OAI-PMH interface, so it'll still exist in the consumer. I should be able to look at my local code today, so I'll look at attaching an alternate patch. (I'll also test this one, because maybe I am misunderstanding this current patch.) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #40 from David Cook <dcook@prosentient.com.au> --- That said, the patch seems to work as described. I'm not too fussed about the "Results fetched 2-50" thing as that's just a human label, and a OAI-PMH consumer shouldn't care that it's only getting 49 records instead of 50. I'm going to look at porting over my local customization and probably will add it as an alternate patch. For now I'll sign off this one though, as I do think it's better than the status quo. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #42 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to David Cook from comment #38)

(In reply to Nick Clemens (kidclamp) from comment #37)

...

We will clearly need tests here, however, the fact that we now respect OpacHiddenItems makes this a bit more urgent - in the past outside systems could use the filtering to determine if a record should be shown, essentially implementing their own OpacHiddenItems - but now the items don't come through and we cannot externally determine if a record truly has no items, or if all items have been hidden.

Marking needs signoff as I need feedback on the approach and assumptions about following 'OpacHiddenItemHidesRecord'

I think that I might like my approach better. If I understand this patch correctly, if a record gets harvested by a consumer, then is hidden in Koha, the record will just disappear from Koha's OAI-PMH interface, so it'll still exist in the consumer.

I should be able to look at my local code today, so I'll look at attaching an alternate patch. (I'll also test this one, because maybe I am misunderstanding this current patch.)

Yeah, I had assumed if you change hiding rules you would want to do a reharvest to catch any that were hidden and no aren't, or vice versa. This actually makes me think the approach of storing the suppressed status at record edit (calculated via a background job) might be better - pre-calculating for search engine and OAI - really you could have OAI fetch via a scroll from ES - though not sure if ES supports resumption - ignore me, ranting I am not sold on these patches, I would love to see the direction you went, this was just the smallest cut along the lines of what you mentioned - determine when fetching record - it's like what we do in the opac - fetch from search engine, then filter. If you don't get to upload your patches I would still be interested just to hear how you approached it. I did consider here returning the hidden records as 'deleted'- it would have been even less code, but that felt like we still told them something was there - rather than full suppress the existence. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #44 from David Cook <dcook@prosentient.com.au> --- Btw, I need to double-check, but I think in your patch you're still showing the records with ListIdentifiers? Your change was just for ListRecord in ListBase.pm -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #46 from David Cook <dcook@prosentient.com.au> --- Created attachment 170300 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170300&action=edit Bug 31161: [Alternate patch] Mark hidden records as deleted If a bib record is hidden by OpacHiddenItems, mark it as deleted in the OAI-PMH, so that it's metadata is not consumed in a new OAI-PMH harvest, and it's marked for removal if the metadata was consumed in previous harvested. Test plan: 0. Setup vi /etc/koha/sites/kohadev/oai.conf format: marcxml: metadataPrefix: marcxml metadataNamespace: http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim schema: http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd include_items: 1 Set sysprefs: OAI-PMH: Enable OAI-PMH:ConfFile: /etc/koha/sites/kohadev/oai.conf OAI-PMH:AutoUpdateSets: Enable OAI-PMH:AutoUpdateSetsEmbedItemData: Enabled OpacHiddenItems: barcode: [999900000001] OpacHiddenItemsHidesRecord: Hide Adminsitration->OAI set configuration->New set Choose anything for setSpec and setName On new set: Actrion->define mappings Field: 942 Subfield: p Operator: is equal to Value: 999900000001 perl misc/migration_tools/build_oai_sets.pl -r -v -i 1. koha-plack --restart kohadev 2. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml 3. Confirm KOHA-OAI-TEST:1 is marked as deleted and not metadata visible 4. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListIdentifiers&metadataPrefix=marcxml 5. Confirm KOHA-OAI-TEST:1 is marked as deleted 6. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=KOHA-OAI-TEST:1 7. Confirm record is marked as deleted and no metadata visible 8. View Sets and confirm the record appears in the set but as deleted record 9. Change OpacHiddenItemsHidesRecord syspref to "Don't hide" and repeat above steps 10. Confirm that the record metadata is now visible and the record is not marked as deleted -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #48 from David Cook <dcook@prosentient.com.au> --- Tempted to move this to "In discussion" but then I know that's where bugs go to die... Jonathan: do you have an opinion which you'd like better? Once Nick fixes the ListIdentifiers and tidy, I think I'd be fine with either approach. I do think mine requires fewer interventions. For instance, I think a lot of libraries would change OpacHiddenItems and not think to redo their OAI-PMH harvests across all consumers. It's a lot of friction. If we did go that route, I think we'd need to update the description for OpacHiddenItems to add a warning that they'll need to advise their OAI-PMH consumers to re-harvest all records each time it's changed. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #50 from David Cook <dcook@prosentient.com.au> --- Jonathan: Turns out the key to getting me to post my patches is for someone else to post a different patch first 🤣 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #52 from Jonathan Field <jonathan.field@ptfs-europe.com> --- (In reply to David Cook from comment #48)

Tempted to move this to "In discussion" but then I know that's where bugs go to die...

Jonathan: do you have an opinion which you'd like better?

Once Nick fixes the ListIdentifiers and tidy, I think I'd be fine with either approach.

I do think mine requires fewer interventions. For instance, I think a lot of libraries would change OpacHiddenItems and not think to redo their OAI-PMH harvests across all consumers. It's a lot of friction.

If we did go that route, I think we'd need to update the description for OpacHiddenItems to add a warning that they'll need to advise their OAI-PMH consumers to re-harvest all records each time it's changed.

I think I'm also fine with either approach. I agree with Nick that, if you were to change opachiddenitems, it wouldn't be unreasonable to expect to do a complete re-harvest. Certainly if EDS was going to be populated from a MARC dump, that's the approach you would take. Having said that, I think marking them as deleted is marginally better in that it's very clear that they are not wanted and it would be easy to spot them and also the consumer could easily act on that too. Happy to test and sign-off either approach. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #53 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens (kidclamp) from comment #51)

Hidden in opac in Koha Biblio doesn't check OpacHiddenItemsHidesRecord, it should. It should also use filter_by_visible rather than looping items, just for simplicity. It should also check if record is suppressed as we're only catching hidden item hiding this way.

Except that Koha::Biblio->hidden_in_opac does check OpacHiddenItemsHidesRecord ;). Good point about filter_by_visible_in_opac. It's a good idea not only because of simplicity but also performance. I think that would be an optimization though. Funny story. In my local version, I do look for OpacSuppression as well. I didn't include it here, because I figured it was out of scope haha. I was going to do it as a follow-up.

Neither of our patches actually triggers a record change, so partial harvests (changes since) won't pickup the difference.

Hmm touché. I suppose that means that the deleted record would show up in subsequent changes... but also all subsequent changes. Hmm... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #55 from Ere Maijala <ere.maijala@helsinki.fi> --- (In reply to David Cook from comment #53)

(In reply to Nick Clemens (kidclamp) from comment #51)

...

Neither of our patches actually triggers a record change, so partial harvests (changes since) won't pickup the difference.

Hmm touché. I suppose that means that the deleted record would show up in subsequent changes... but also all subsequent changes. Hmm...

Just as a sidenote: I believe the proper way around this would be a publishing process that would periodically publish a new set of records. It could compare the new set against the previous one and add any records no longer available as deleted (it would of course need to carry any previously deleted records over as well, but it could keep their old timestamps). The publishing process would be more complicated, but then the actual OAI-PMH provider (replies to harvesting requests) could be really simple and fast. This would have the additional benefit that the set of records being harvested would never change in the middle of the harvest. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 Jake Deery <jake.deery@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170280|0 |1 is obsolete| | --- Comment #57 from Jake Deery <jake.deery@ptfs-europe.com> --- Created attachment 170353 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170353&action=edit Bug 31161: Suppress OAI records when all items hidden by OpacHiddenItems This patch set adds an option to Koha::Metadata::Record to suppress a record if all items are hidden. This option is only enabled by OAI and only when OpacHiddenItemsHidesRecord is enabled. This test plan assumes KTD default data To setup: edit/create /etc/koha/sites/kohadev/oai.conf with content below: format: marcxml: metadataPrefix: marcxml metadataNamespace: http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim schema: http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd include_items: 1 Set sysprefs: OAI-PMH: Enable OAI-PMH:ConfFile: /etc/koha/sites/kohadev/oai.conf OAI-PMH:AutoUpdateSets: Enable OAI-PMH:AutoUpdateSetsEmbedItemData: Enabled OpacHiddenItems: itype: [BK] OpacHiddenItemsHidesRecord: Hide Adminsitration->OAI set configuration->New set Choose anything for setSpec and setName On new set: Actrion->define mappings Field: 942 Subfield: c Operator: is equal to Value: MU To test: 1 - Set above, do not apply patch, restart_all 2 - Visit: http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml 3 - Confirm record 1 is visible wit no items (or anothe record that has BK itemtypes only) 4 - perl misc/migration_tools/build_oai_sets.pl -r -v -i 5 - Confirm 11 records in set created above (or correct number for your site) 6 - Apply patches, restart_all 7 - Repeat 2, note "Results fetched" is 43-50, count is off because of hidden records 8 - Confirm record 1 and many others not shown 9 - Confirm 'Show more' works 10 - Repeat 4, still 11 11 - Find a record with record type music, say 315 12 - Edit all items and change type to BK - Books 13 - Repeat 4 - now 10 records 14 - Set pref OpacHiddenItemsHidesRecord: Don't hide 15 - Repeat 2 and 4 and confirm results as before patch Signed-off-by: Jake Deery <jake.deery@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 Jake Deery <jake.deery@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170301|0 |1 is obsolete| | --- Comment #59 from Jake Deery <jake.deery@ptfs-europe.com> --- Created attachment 170354 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170354&action=edit Bug 31161: [Alternate patch] Mark hidden records as deleted If a bib record is hidden by OpacHiddenItems, mark it as deleted in the OAI-PMH, so that it's metadata is not consumed in a new OAI-PMH harvest, and it's marked for removal if the metadata was consumed in previous harvested. Test plan: 0. Setup vi /etc/koha/sites/kohadev/oai.conf format: marcxml: metadataPrefix: marcxml metadataNamespace: http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim schema: http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd include_items: 1 Set sysprefs: OAI-PMH: Enable OAI-PMH:ConfFile: /etc/koha/sites/kohadev/oai.conf OAI-PMH:AutoUpdateSets: Enable OAI-PMH:AutoUpdateSetsEmbedItemData: Enabled OpacHiddenItems: barcode: [3999900000001] OpacHiddenItemsHidesRecord: Hide Adminsitration->OAI set configuration->New set Choose anything for setSpec and setName On new set: Actrion->define mappings Field: 952 Subfield: p Operator: is equal to Value: 3999900000001 perl misc/migration_tools/build_oai_sets.pl -r -v -i 1. koha-plack --restart kohadev 2. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml 3. Confirm KOHA-OAI-TEST:1 is marked as deleted and not metadata visible 4. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListIdentifiers&metadataPrefix=marcxml 5. Confirm KOHA-OAI-TEST:1 is marked as deleted 6. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=KOHA-OAI-TEST:1 7. Confirm record is marked as deleted and no metadata visible 8. View Sets and confirm the record appears in the set but as deleted record 9. Change OpacHiddenItemsHidesRecord syspref to "Don't hide" and repeat above steps 10. Confirm that the record metadata is now visible and the record is not marked as deleted Signed-off-by: Jake Deery <jake.deery@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #61 from David Cook <dcook@prosentient.com.au> --- (In reply to Jake Deery from comment #60)

1) I had to update the barcode from 999900000001 to 3999900000001 — I think this was probably a typo, as 3999900000001 is the barcode for bib #1 2) I also had to update the OAI set MARC field from 942 to 952, because that's where the item data is kept. 942$p didn't work.

Ooops sorry! -- Jake, did you test ListIdentifiers with Nick's patch? I don't think it's been fixed yet. I don't want to Fail QA, but could you take a look at that one? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #63 from David Cook <dcook@prosentient.com.au> --- (In reply to Jake Deery from comment #62)

Discuss!!

I think Nick was out of town for a few days, so may as well wait for him to send an updated patch? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170353|0 |1 is obsolete| | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #66 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens (kidclamp) from comment #65)

David, looks like you have a #FIXME and a #TODO - are those to be handled here?

The #TODO will be a follow-up once this is in. I can open it now and just put a "Depends on". The "FIXME" is more a question to QAers/RM. Typically we try not to change existing behaviour with new changes, so it makes me wonder if we would want to syspref it. Locally, we have a syspref around it, which lets instances opt out of this behaviour, but I don't think I've had anyone opt out of it. Everyone has wanted it. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37713 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37713 [Bug 37713] OAI-PMH - Honour OpacSuppression syspref -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Passed QA -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #69 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 170633 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170633&action=edit Bug 31161: (QA follow-up) Remove FIXME Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |nick@bywatersolutions.com |y.org | Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Failed QA Keywords| |release-notes-needed --- Comment #70 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Since there was a bit of back and forth on this one, please detail/summarize in release notes! Please add unit tests. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #71 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 171074 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171074&action=edit Bug 31161: Unit tests -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | Text to go in the|This patch alters the |This patch alters the release notes|OAI-PMH code to respect the |OAI-PMH code to respect the |OpacHiddenItems system |OpacHiddenItems and |preference when all items |OpacHiddenItemsHidesRecord |on a record are hidden. In |system preferences when all |this case, the server will |items on a record are |now return the record as |hidden. In this case, the |'deleted' - this way if an |server will now return the |item is changed in a way |record as 'deleted' - this |that marks it hidden, and |way if an item is changed |it is the last item on the |in a way that marks it |record, the next harvest |hidden, and it is the last |will pickup this change and |item on the record, the |remove the record |next harvest will pickup | |this change and remove the | |record --- Comment #72 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Looking at the code of this I was missing a reference to OpacHiddenItemsHidesRecord, but none the less... it works. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31161 --- Comment #73 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.