I’ve whipped up some code and posted it at <http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15537> http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15537. It should work perfectly for MARC21 after applying the patch and doing a full re-index of Zebra. I’ve provided changes for NORMARC and UNIMARC as well, but I haven’t tested them. Cheers, David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007 From: Brendan Gallagher [mailto:info@bywatersolutions.com] Sent: Tuesday, 12 January 2016 4:59 AM To: David Cook <dcook@prosentient.com.au> Cc: Koha Devel <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Hide records on Leader 05 = d in OPAC Sounds good to me David. If you get some code up there I'll find someone to Test. Cheers, Brendan On Sunday, January 10, 2016, David Cook <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > wrote: Hi all: Recently, http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11084 was pushed to master. It adds a cronjob which deletes bibliographic records, if their LDR05 is “d” (ie status deleted). While I’m all for that, I’m thinking that it would also be a good idea to hide these records with LDR05 “d” in the OPAC before the cronjob is run. We can’t necessarily rely on all Koha instances running this cronjob, nor can we rely on the frequency. Shouldn’t we be hiding these records from the OPAC as soon as they’re marked as “deleted”? I’ve opened a bug for this purpose: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15537 It should be a pretty easy change, especially when using special attribute @attr 14=1 to avoid past OpacSuppression problems, but I thought I’d ask folk if it’s something they’d be interested in. We’d probably still want to see these records in the Staff Client, as we want to be able to find all records in the database in the Staff Client. I admit that I have a special interest in this where I might be overlaying existing records using a mostly empty skeleton record generated from an OAI-PMH identifier and a OAI-PMH deleted status (OAI-PMH doesn’t send metadata for deleted records). I’d match the existing record in Koha using the identifier, and then set LDR05 to “d” in accordance with the OAI-PMH deleted status. Then, that record would disappear from the OPAC, so that end users don’t see this skeleton record. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007 -- Sent from Gmail Mobile

From a Koha db viewpoint, either the record is in MySQL and therefore (probably?) should get indexed by Zebra, or it has been deleted by a cataloguer -- logically gone, terminally, no bytes left on the hard disk -- except that Koha has tables "deletedbiblio", "deletedbiblioitems" and "deleteditems" which (as far as I can tell) do not contain the LDR,

I imagine that most cataloguers cataloguing in Koha would just hit "Delete record" in Koha. However, if you're cataloguing outside of Koha, you can modify LDR05 to "d", and then import a batch of MARC records into Koha. When you overlay the records in Koha, they would conceptually be deleted, so a cronjob could come and "reap" them later without any additional manual intervention. My work on <http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15537> http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15537 will prevent these records from showing up in the OPAC in the interim as well. That said, there is a question as to why we're saving records with a LDR05 of "d" in the database. In theory, if LDR05=d, we could just delete the record as soon as we see that. However, there are other considerations to make such as item-level data, subscription data, etc. That's a bit out of the scope of my caring at the moment. I'm working on batch importing of records, so I care about being able to signal to Koha that I want to add some records, update some records, and delete other records. As for "undeleting", I don't think anyone is currently working on that, but I'm sure someone could. Historically, I think the LDR05=d was used to remove records from the OPAC but still let staff see them for a certain period of time before they were removed. I think removal may have even been configurable. I've been working on Koha long enough to not recall 100% how other ILSes worked in that respect in the past. For what it's worth, the leader is stored in the deletedbiblioitems.marcxml as well. I think it's a perfect copy of the record at the point in time of its deletion. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007 From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Paul A Sent: Tuesday, 12 January 2016 11:57 AM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Hide records on Leader 05 = d in OPAC At 12:44 PM 1/11/2016 +1100, David Cook wrote: Content-Type: multipart/alternative; boundary="----=_NextPart_000_008A_01D14C6D.DDE56680" Content-Language: en-au Hi all: Recently, http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11084 was pushed to master. It adds a cronjob which deletes bibliographic records, if their LDR05 is d(ie status deleted). Brendan, David, Thanks for your work on this. I have sometimes wondered what the LDR, Pos 05 - Record status d - Deleted "Record has been deleted" actually means. My question is: how many cataloguers take the time and trouble to modify LDR pos05, rather than just "hit delete" for a record? therefore cannot cross-reference pos05=d (and I'm not certain that there is a way to "undelete" -- can it be done?) Best -- Paul While Im all for that, Im thinking that it would also be a good idea to hide these records with LDR05 din the OPAC before the cronjob is run. We cant necessarily rely on all Koha instances running this cronjob, nor can we rely on the frequency. Shouldnt we be hiding these records from the OPAC as soon as theyre marked as deleted? Ive opened a bug for this purpose: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15537 It should be a pretty easy change, especially when using special attribute @attr 14=1 to avoid past OpacSuppression problems, but I thought Id ask folk if its something theyd be interested in. Wed probably still want to see these records in the Staff Client, as we want to be able to find all records in the database in the Staff Client. I admit that I have a special interest in this where I might be overlaying existing records using a mostly empty skeleton record generated from an OAI-PMH identifier and a OAI-PMH deleted status (OAI-PMH doesnt send metadata for deleted records). Id match the existing record in Koha using the identifier, and then set LDR05 to din accordance with the OAI-PMH deleted status. Then, that record would disappear from the OPAC, so that end users dont see this skeleton record. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/ --- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com <http://navalmarinearchive.com/> > and <http://UltraMarine.ca <http://ultramarine.ca/> >

On Tue, Jan 12, 2016 at 08:48:21AM -0500, Galen Charlton wrote:

Hi,

On Sun, Jan 10, 2016 at 8:44 PM, David Cook <dcook@prosentient.com.au> wrote:

...

We can’t necessarily rely on all Koha instances running this cronjob, nor can we rely on the frequency. Shouldn’t we be hiding these records from the OPAC as soon as they’re marked as “deleted”?

I’ve opened a bug for this purpose: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15537

I am in mild disfavor of this proposal, particularly as implemented in current patch. Using a cronjob to delete records where Leader/05 is set to 'd' is useful when the library has arranged their workflow such that they *know* that Leader/05 = 'd' is being used consistently to signify that a record is no longer wanted. However, for a library that has not hitherto cared about the values in that position, unconditionally suppressing the display of such records could come as an unwelcome surprise.

I'd echo Galen's caution, especially if records are being imported, I've seen cases where trying to do this has revealed that records in use locally have inherited a deleted status from an upstream provider and does not reflect local status. Colin -- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2 http://www.ptfs-europe.com

Hi Galen: I think adding a configuration option makes sense in that case. The thought occurred to me at the time, but I figured it would be ubiquitously welcome without one. A configuration option is easy enough to add. Adding a mechanism for specifying a set of query additions could be interesting, although it would involve some refactoring and re-testing of existing Koha code, which I doubt anyone would want to do.

I do not find this a compelling use case as stated. If the goal is to allow harvesting and overlay records from an OAI-PMH provider to also delete bibs from a Koha database... coding so that the records are *actually* deleted seems more direct.

Originally, I thought the same thing, Galen. I thought that I would download the OAI-PMH data, see the deleted status, and then just delete the record from Koha. However, I soon realized that I actually didn't have any way of doing that. The OAI-PMH data only contains the deleted status and the unique OAI-PMH identifier. The update is coming from upstream and it has no knowledge of downstream. So I first need to match that unique OAI-PMH identifier with a record in Koha before I can directly delete the record. At this point, I think the consensus amongst community members is to use Zebra, the record matching rules, and the MARC import_bib API for ingesting MARC records programmatically. And that works really well for additions and updates. However, it doesn't work so well for deletions. In theory, we could try to make it delete a record as soon as it sees a Leader/05=d, but that runs into the risks you and Colin described where librarians might not realize that Leader/05=d is set. I suppose another option could be added that makes this more explicit, but then we're changing an API which has been in steady use for years. I suppose I could keep a table tying the OAI-PMH identifier to a Koha biblionumber, but that's error-prone (especially due to bibliographic record merges, local deletions, etc) and it's a lot of data to keep for the single purpose of deleting records. Additionally, processing the deletions directly is inherently problematic. If there are items, the deletion will fail. I think there might be a few other foreign relationships which will cause bibliographic record deletes to fail. At that point, the third-party is at an impasse. They might know that they have failed to delete a record in Koha, but no one in Koha knows that a record wasn't successfully deleted. Going back to the added table idea, I suppose I could store the OAI-PMH identifier, and note an error if it fails to delete, and then Koha users can check that. However, that adds more overhead to the import of MARC records via an OAI-PMH provider. I'd have to make another web service to update the OAI-PMH table before trying to send it to import_bib. That's double the HTTP requests though, so maybe I'd just make import_oai and skip import_bib all together, but then I'm duplicating most of the code in import_bib just to support deletions. That seems inefficient and error-prone as well. I'm reflecting on the wording in your last message, Galen. I suppose the main goal is to harvest records from an OAI-PMH server, and then to add/update MARC records in Koha. That works well. But a secondary goal is to delete records from Koha where they've been deleted upstream in the OAI-PMH server, yes. As I noted above though, upstream doesn't have knowledge of downstream. I suppose I could query Zebra myself using Z39.50 or SRU (although SRU is disabled by default I believe), but then I lose the record matching rules, which are vital to matching MARC record fields with Zebra indexes. But let's say I query Zebra directly, I could then get the biblionumber I need and then use "/svc/bib" to attempt a deletion. Although that still runs into the same problem as before... the automatic deletion might fail... and no one in Koha will know. At that point, I suppose I could try sending an email, or have some other error reporting mechanism to Koha users... but then the added code is simply to explain that a deletion has failed. It seems simpler and less error-prone to mark a record as deleted, hide it from the OPAC, and then let Koha users either use searches or reports to find Leader/05=d records to deal with according to their own discretion. I suppose the reports would be difficult on larger databases, but a search would work well. "Record-status=d" would show all records listed as deleted with reasonable efficiency. I like the idea of just deleting records directly, but I think it's more complex than it appears at first glance. It's not just an issue with OAI-PMH either really. It's an issue any time you try to delete a record without providing feedback to an end-user. I am open to suggestions though. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007

-----Original Message----- From: Galen Charlton [mailto:gmc@esilibrary.com] Sent: Wednesday, 13 January 2016 12:48 AM To: David Cook <dcook@prosentient.com.au> Cc: Koha Devel <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Hide records on Leader 05 = d in OPAC

Hi,

On Sun, Jan 10, 2016 at 8:44 PM, David Cook <dcook@prosentient.com.au> wrote:

...

We can’t necessarily rely on all Koha instances running this cronjob, nor can we rely on the frequency. Shouldn’t we be hiding these records from the OPAC as soon as they’re marked as “deleted”?

I’ve opened a bug for this purpose: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15537

I am in mild disfavor of this proposal, particularly as implemented in current patch. Using a cronjob to delete records where Leader/05 is set to 'd' is useful when the library has arranged their workflow such that they *know* that Leader/05 = 'd' is being used consistently to signify that a record is no longer wanted. However, for a library that has not hitherto cared about the values in that position, unconditionally suppressing the display of such records could come as an unwelcome surprise.

That said, it is also a reasonable choice for a library to want to use the Leader/05 as suppression criterion. Consequently, I suggest adding a configuration option. For that matter, making it configurable (say, by allowing the library to specify a set of query additions for the purpose of filtering records from public display) could result in a more generally useful mechanism.

...

I admit that I have a special interest in this where I might be overlaying existing records using a mostly empty skeleton record generated from an OAI-PMH identifier and a OAI-PMH deleted status (OAI-PMH doesn’t send metadata for deleted records). I’d match the existing record in Koha using the identifier, and then set LDR05 to “d” in accordance with the OAI-PMH deleted status. Then, that record would disappear from the OPAC, so that end users don’t see this skeleton record.

I do not find this a compelling use case as stated. If the goal is to allow harvesting and overlay records from an OAI-PMH provider to also delete bibs from a Koha database... coding so that the records are *actually* deleted seems more direct.

Regards,

Galen -- Galen Charlton Infrastructure and Added Services Manager Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org

Hi, For this specific project, I personally don't have control of anything per se, but the sponsors of the code will have control of their Koha instance and theoretically the data in the upstream data provider - although not control of the data provider itself. I have thought a bit about policy, but I'm not sure that it's entirely relevant in this case. Regardless of ownership, you can't really do a cascade delete if a bib's items are on loan or on hold or otherwise engaged. For example, let's say that the upstream data provider was the sole provider of bib and item data into Koha. If they delete the record upstream but an item is engaged downstream, the delete will fail. I suppose in my particular case it might not matter too much... as the library might not delete their records from upstream until the items have been dealt with downstream. But I doubt that would be the case for all users and it doesn't account for user mistakes either. You could still end up with orphaned records in Koha that no one necessarily knows to remove. As for the scenario where the Koha user can add items to harvested bibs, which may be the case for the sponsors, that is indeed more complicated. On one hand, the upstream data provider provides all the bibliographic data so they "own" the record. On the other hand, does the upstream data provider have knowledge of the downstream items? They might or they might not... I think perhaps I'm beginning to understand your point. Due to the variety of scenarios, perhaps it makes sense to track all OAI-PMH transactions, and then have policies for controlling how deletions are processed from there. That way, even if direct deletions are the norm, errors for failed deletions can be tracked. Or if users would prefer to know that an upstream bib has been deleted but they want to keep their downstream bib, that can be an option too. Those options can also be fleshed out over time as needed in terms of a dashboard, alerting, etc. Tracking the transactions in the database would help me prevent some race conditions for asynchronous importing as well... Hmm, yes, I can see the sense of preventing that spill over beyond the OAI-PMH import domain. I think perhaps my original proposal was trying to cut too many corners. I'm tempted to amend the Bugzilla patch to include a system preference, but I suspect that I won't end up using this in my project anymore. Thanks for your feedback, Galen. It's much appreciated. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007

-----Original Message----- From: Galen Charlton [mailto:gmc@esilibrary.com] Sent: Saturday, 16 January 2016 2:36 AM To: David Cook <dcook@prosentient.com.au> Cc: Koha Devel <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Hide records on Leader 05 = d in OPAC

Hi,

On Fri, Jan 15, 2016 at 12:46 AM, David Cook <dcook@prosentient.com.au> wrote:

...

I like the idea of just deleting records directly, but I think it's more complex than it appears at first glance. It's not just an issue with OAI-PMH either really. It's an issue any time you try to delete a record without providing feedback to an end-user.

I've got a question for you: for the specific project you're coding for, which ends do you control? The Koha instance, the data provider publishing via OAI-PMH, or both?

To make a general point: yep, there are definitely edge cases and error conditions to consider when implementing a mechanism by which a third party can specify that records should be deleted in a Koha database. Some of those might be better solved by policy rather than code; for example, if the OAI-PMH provider in some sense "owns" the records that Koha ingests from it, does the Koha library need to a have policy of not adding items to such records? If so, it might be appropriate to add an option that specifies that bib deletions are to forcibly cascade.

Conversely, if it *is* legitimate for the Koha user to add items to those records, does that mean that the OAI-PMH provider no longer has "ownership"?

To make another general point: I think it would be better for the consequences of record deletion to be handled *within the context of OAI- PMH harvesting* (or more generally, mechanisms to sync records with an external provider), but *not* to have those consequences spill over for users who are not doing such harvesting as all. Your original proposal to unconditionally hide Leader/05='d' records from the public catalog would be an example of such spillover.

Regards,

Galen -- Galen Charlton Infrastructure and Added Services Manager Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org