MARC::Record, record length and leader
Hello, I know this is an old discussion but I just realized that saving a biblio record longer than 99999 bytes result in a malformed leader. Length of the record takes more than 5 characters and therefore remaining characters in the leader are shifted to the right. This cause the leader to have more than 24 characters. I'm not sure of what have to be done. Should MARC::Record die when encountering such a record? or should it force length to be 99999? or should the record be truncated to fit the spec? Has someone ever faced this problem? How did you handle it? -- Julian Maurice <julian.maurice@biblibre.com> BibLibre
Julian, I know this is an old discussion but I just realized that saving a biblio
record longer than 99999 bytes result in a malformed leader. Length of the record takes more than 5 characters and therefore remaining characters in the leader are shifted to the right. This cause the leader to have more than 24 characters.
I'm not sure of what have to be done. Should MARC::Record die when encountering such a record? or should it force length to be 99999? or should the record be truncated to fit the spec? Has someone ever faced this problem? How did you handle it?
MARC::Record should proceed blithely on, like Admiral Farragut at the Battle of Mobile Bay or the Light Brigade at the Battle of Balaclava. The record can still be serialized as MARCXML, and large records are not uncommon in the wild. As far as I know, the only way to handle it is to use MARCXML instead of iso2709. Regards, Jared -- Jared Camins-Esakov Bibliographer, C & P Bibliography Services, LLC (phone) +1 (917) 727-3445 (e-mail) jcamins@cpbibliography.com (web) http://www.cpbibliography.com/
On 10/04/2012 16:27, Jared Camins-Esakov wrote:
Julian,
I know this is an old discussion but I just realized that saving a biblio record longer than 99999 bytes result in a malformed leader. Length of the record takes more than 5 characters and therefore remaining characters in the leader are shifted to the right. This cause the leader to have more than 24 characters.
I'm not sure of what have to be done. Should MARC::Record die when encountering such a record? or should it force length to be 99999? or should the record be truncated to fit the spec? Has someone ever faced this problem? How did you handle it?
MARC::Record should proceed blithely on, like Admiral Farragut at the Battle of Mobile Bay or the Light Brigade at the Battle of Balaclava. The record can still be serialized as MARCXML, and large records are not uncommon in the wild. As far as I know, the only way to handle it is to use MARCXML instead of iso2709.
Regards, Jared
Ok, but the malformed leader is saved in the MARCXML, so it's displayed in the MARC editor, and cataloguing plugins marc21_leader.pl and unimarc_leader.pl won't work correctly. Isn't there any workaround for this? More than that, Koha is still generating malformed MARC records, and I think it's a problem for an ILS which is MARC-compliant. -- Julian Maurice <julian.maurice@biblibre.com> BibLibre
Julian, MARC::Record should proceed blithely on, like Admiral Farragut at the
Battle of Mobile Bay or the Light Brigade at the Battle of Balaclava. The record can still be serialized as MARCXML, and large records are not uncommon in the wild. As far as I know, the only way to handle it is to use MARCXML instead of iso2709.
Ok, but the malformed leader is saved in the MARCXML, so it's displayed in the MARC editor, and cataloguing plugins marc21_leader.pl and unimarc_leader.pl won't work correctly. Isn't there any workaround for this?
Not to my knowledge.
More than that, Koha is still generating malformed MARC records, and I think it's a problem for an ILS which is MARC-compliant.
It's not ideal, certainly, but Koha is using internally a format (i.e. MARC) that isn't really suitable for the twenty-first century. As far as malformed MARC records go, Koha's offenses are much less egregious than certain other ILSes'. Regards, Jared -- Jared Camins-Esakov Bibliographer, C & P Bibliography Services, LLC (phone) +1 (917) 727-3445 (e-mail) jcamins@cpbibliography.com (web) http://www.cpbibliography.com/
I'd advocate ISO-2709 only being an import/export format, and never using it internally. It's just got too many limitations, as we're seeing here. On import, we check for valid leaders, and warn if not. On export, we calculate the leader, and if it exceeds 99999, we warn. We can develop several options for dealing with this, like truncating to the appropriate length, skipping the record or just exporting with an invalid leader. Cheers, -Ian On Tue, Apr 10, 2012 at 11:05, Julian Maurice <julian.maurice@biblibre.com>wrote:
On 10/04/2012 16:27, Jared Camins-Esakov wrote:
Julian,
I know this is an old discussion but I just realized that saving a biblio record longer than 99999 bytes result in a malformed leader. Length of the record takes more than 5 characters and therefore remaining characters in the leader are shifted to the right. This cause the leader to have more than 24 characters.
I'm not sure of what have to be done. Should MARC::Record die when encountering such a record? or should it force length to be 99999? or should the record be truncated to fit the spec? Has someone ever faced this problem? How did you handle it?
MARC::Record should proceed blithely on, like Admiral Farragut at the Battle of Mobile Bay or the Light Brigade at the Battle of Balaclava. The record can still be serialized as MARCXML, and large records are not uncommon in the wild. As far as I know, the only way to handle it is to use MARCXML instead of iso2709.
Regards, Jared
Ok, but the malformed leader is saved in the MARCXML, so it's displayed in the MARC editor, and cataloguing plugins marc21_leader.pl and unimarc_leader.pl won't work correctly. Isn't there any workaround for this?
More than that, Koha is still generating malformed MARC records, and I think it's a problem for an ILS which is MARC-compliant.
-- Julian Maurice <julian.maurice@biblibre.com> BibLibre ______________________________**_________________ Koha-devel mailing list Koha-devel@lists.koha-**community.org<Koha-devel@lists.koha-community.org> http://lists.koha-community.**org/cgi-bin/mailman/listinfo/**koha-devel<http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.**org/ <http://bugs.koha-community.org/>
Le 10/04/2012 17:15, Ian Walls a écrit :
I'd advocate ISO-2709 only being an import/export format, and never using it internally. It's just got too many limitations, as we're seeing here.
On import, we check for valid leaders, and warn if not.
On export, we calculate the leader, and if it exceeds 99999, we warn. We can develop several options for dealing with this, like truncating to the appropriate length, skipping the record or just exporting with an invalid leader.
Cheers,
-Ian
It's not really about ISO-2709 format (afaik, MARCXML has the same problem, leader must be 24 characters long), it's about MARC. And we are using it internally through MARC::Record. As we already have invalid MARC records (ie. longer than 99999 bytes), and we want to keep the records as is, I think we should at least have a well-formed (but invalid) leader with first five characters being '99999', since sixth character will never be read to get the record length anyway. What do you think? -- Julian Maurice <julian.maurice@biblibre.com> BibLibre
Le 11/04/2012 09:40, Julian Maurice a écrit :
It's not really about ISO-2709 format (afaik, MARCXML has the same problem, leader must be 24 characters long), it's about MARC. And we are using it internally through MARC::Record. As we already have invalid MARC records (ie. longer than 99999 bytes), and we want to keep the records as is, I think we should at least have a well-formed (but invalid) leader with first five characters being '99999', since sixth character will never be read to get the record length anyway. What do you think?
In order to close this discussion, I created a ticket on rt.cpan.org (https://rt.cpan.org/Public/Bug/Display.html?id=76990) and attached a patch to force record length to not be greater than 99999 in the leader. -- Julian Maurice <julian.maurice@biblibre.com> BibLibre
Hi, On May 4, 2012, at 10:38 AM, Julian Maurice wrote:
In order to close this discussion, I created a ticket on rt.cpan.org (https://rt.cpan.org/Public/Bug/Display.html?id=76990) and attached a patch to force record length to not be greater than 99999 in the leader.
This would have the effect of having MARC::Record output invalid records that we *know* are invalid. There's no way around it -- an ISO2709 blob can be no longer than 99999 octets, and a strict parser (a couple of which do exist in the wild) would drop records where the Leader/05 does not match the length of the record as determined by the location of the record terminator character. I'm much more inclined to accept a patch to MARC::Record that introduces a more forgiving input parsing mode. Regards, Galen -- Galen Charlton Director of Support and Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
Le 07/05/2012 17:30, Galen Charlton a écrit :
Hi,
Hi!
On May 4, 2012, at 10:38 AM, Julian Maurice wrote:
In order to close this discussion, I created a ticket on rt.cpan.org (https://rt.cpan.org/Public/Bug/Display.html?id=76990) and attached a patch to force record length to not be greater than 99999 in the leader.
This would have the effect of having MARC::Record output invalid records that we *know* are invalid. There's no way around it -- an ISO2709 blob can be no longer than 99999 octets, and a strict parser (a couple of which do exist in the wild) would drop records where the Leader/05 does not match the length of the record as determined by the location of the record terminator character.
Actually, MARC::Record will always output invalid records if they're longer than 99999 bytes. But with the patch, at least the leader is well-formed (24 chars long).
I'm much more inclined to accept a patch to MARC::Record that introduces a more forgiving input parsing mode.
What do you mean by "a more **forgiving** input parsing mode" ? -- Julian Maurice <julian.maurice@biblibre.com> BibLibre
I think there is not a more forgiving parser. The ISO2709 spec breaks when a record reaches 100000 octets. I don't see any way around that in the parser. The best solution I can come up with, if you want to hold to the spec, is to create a seperate ISO2709 blob with a logical link to the first when a record reaches 100000 octets. I don't know if that could be implemented in the parser though. I think that any other way will result in either a broken record or a broken parser. On Wed, May 9, 2012 at 1:23 AM, Julian Maurice <julian.maurice@biblibre.com>wrote:
Le 07/05/2012 17:30, Galen Charlton a écrit :
Hi,
Hi!
On May 4, 2012, at 10:38 AM, Julian Maurice wrote:
In order to close this discussion, I created a ticket on rt.cpan.org ( https://rt.cpan.org/Public/**Bug/Display.html?id=76990<https://rt.cpan.org/Public/Bug/Display.html?id=76990>) and attached a patch to force record length to not be greater than 99999 in the leader.
This would have the effect of having MARC::Record output invalid records that we *know* are invalid. There's no way around it -- an ISO2709 blob can be no longer than 99999 octets, and a strict parser (a couple of which do exist in the wild) would drop records where the Leader/05 does not match the length of the record as determined by the location of the record terminator character.
Actually, MARC::Record will always output invalid records if they're longer than 99999 bytes. But with the patch, at least the leader is well-formed (24 chars long).
I'm much more inclined to accept a patch to MARC::Record that introduces a more forgiving input parsing mode.
What do you mean by "a more **forgiving** input parsing mode" ?
-- Julian Maurice <julian.maurice@biblibre.com> BibLibre ______________________________**_________________ Koha-devel mailing list Koha-devel@lists.koha-**community.org<Koha-devel@lists.koha-community.org> http://lists.koha-community.**org/cgi-bin/mailman/listinfo/**koha-devel<http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.**org/ <http://bugs.koha-community.org/>
Hie, That's right, Koha uses both marc and marcxml for data storage. It costs performance on insert and update. Can we imagine dropping marc field form database and serialize on demande ? -- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France 2012/4/10 Ian Walls <koha.sekjal@gmail.com>
I'd advocate ISO-2709 only being an import/export format, and never using it internally. It's just got too many limitations, as we're seeing here.
On import, we check for valid leaders, and warn if not.
On export, we calculate the leader, and if it exceeds 99999, we warn. We can develop several options for dealing with this, like truncating to the appropriate length, skipping the record or just exporting with an invalid leader.
Cheers,
-Ian
On Tue, Apr 10, 2012 at 11:05, Julian Maurice <julian.maurice@biblibre.com
wrote:
On 10/04/2012 16:27, Jared Camins-Esakov wrote:
Julian,
I know this is an old discussion but I just realized that saving a biblio record longer than 99999 bytes result in a malformed leader. Length of the record takes more than 5 characters and therefore remaining characters in the leader are shifted to the right. This cause the leader to have more than 24 characters.
I'm not sure of what have to be done. Should MARC::Record die when encountering such a record? or should it force length to be 99999? or should the record be truncated to fit the spec? Has someone ever faced this problem? How did you handle it?
MARC::Record should proceed blithely on, like Admiral Farragut at the Battle of Mobile Bay or the Light Brigade at the Battle of Balaclava. The record can still be serialized as MARCXML, and large records are not uncommon in the wild. As far as I know, the only way to handle it is to use MARCXML instead of iso2709.
Regards, Jared
Ok, but the malformed leader is saved in the MARCXML, so it's displayed in the MARC editor, and cataloguing plugins marc21_leader.pl and unimarc_leader.pl won't work correctly. Isn't there any workaround for this?
More than that, Koha is still generating malformed MARC records, and I think it's a problem for an ILS which is MARC-compliant.
-- Julian Maurice <julian.maurice@biblibre.com> BibLibre ______________________________**_________________ Koha-devel mailing list Koha-devel@lists.koha-**community.org<Koha-devel@lists.koha-community.org> http://lists.koha-community.**org/cgi-bin/mailman/listinfo/**koha-devel<http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.**org/<http://bugs.koha-community.org/>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
<fridolyn.somers@gmail.com>
Partha Mukhopadhyay <psm_vu@india.com>
Koha does not include (upto version 3.6.4) the provision of MARC 21 authority management from GUI interfaces. For example 1) to import a MARC formatted authority file (like bibliographic records), 2) to export authority records from local records through GUI etc. Presently it provides only a script (bulkauthimport.pl) to import MARC 21 authority records. Could you please suggest me 1) way to export of local authority data as marc file; 2) process to setup a Z 39.50 authority data server (over the port 9999) thorugh Koha? If this version (3.6.4) cannot do, are these features planned for 3.8 or 3.10 series?
1. I'm not sure. I don't think it would be too difficult to extend tools/export.pl but I could be wrong. There is bug http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=5706 tracking this feature request, but no library has offered to sponsor it yet. http://wiki.koha-community.org/wiki/Bug_Reporting_Guidelines#Enhancement.2FS... 2. This can be done by adding or uncommenting lines in koha-conf.xml as described in the Zebrasrv manual to set up another virtual server. http://www.indexdata.com/zebra/doc/zebrasrv.html#gfs-config Next time, would you mind making sure you use the "New Mail" command to start a new thread for a new topic, rather than replying to an old one? If you don't, it gets sent with the wrong metadata so the web forums and some email clients will file it under the wrong thread, which means the emails are less likely to be read by the right people. Thanks, -- MJ Ray (slef), member of www.software.coop, a for-more-than-profit co-op. http://koha-community.org supporter, web and library systems developer. In My Opinion Only: see http://mjr.towers.org.uk/email.html Available for hire (including development) at http://www.software.coop/
On 11/04/12 19:43, MJ Ray wrote: > Partha Mukhopadhyay<psm_vu@india.com> >> Koha does not include (upto version 3.6.4) the provision of MARC 21 >> authority management from GUI interfaces. For example 1) to import a >> MARC formatted authority file (like bibliographic records), 2) to >> export authority records from local records through GUI >> etc. Presently it provides only a script (bulkauthimport.pl) to >> import MARC 21 authority records. Could you please suggest me 1) way >> to export of local authority data as marc file; 2) process to setup >> a Z 39.50 authority data server (over the port 9999) thorugh Koha? >> If this version (3.6.4) cannot do, are these features planned for >> 3.8 or 3.10 series? > 1. I'm not sure. I don't think it would be too difficult to extend > tools/export.pl but I could be wrong. There is bug > http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=5706 > tracking this feature request, but no library has offered to sponsor it > yet. http://wiki.koha-community.org/wiki/Bug_Reporting_Guidelines#Enhancement.2FSponsorship_Guidelines Actually, the more relevant enhancement for importing authorities is bug 2060: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=2060 Nicole raised the question in 2008! I can't believe librarians are not screaming the house down to have this feature added. But maybe its not an issue for most people? Or the libraries that need it are totally impecunious? Just wondering really. Bob Calyx
Is the record length portion of the leader really useful to anyone anymore? It's really a much more valuable concept 30 years ago, when storage in bytes was a real consideration. I'd say just stop counting at 99999, and let the export scripts provide the recalculation, truncation or warning. Since we don't include items internally in our MARC except for indexing, we shouldn't run into this situation very often now. Novel idea: what about allowing hexidecimal values for those 5 characters? That'd give us an extra 948577 bytes, which would cover most cases, I'd think. -Ian 2012/4/11 Fridolyn SOMERS <fridolyn.somers@gmail.com>
Hie,
That's right, Koha uses both marc and marcxml for data storage. It costs performance on insert and update.
Can we imagine dropping marc field form database and serialize on demande ?
-- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France
2012/4/10 Ian Walls <koha.sekjal@gmail.com>
I'd advocate ISO-2709 only being an import/export format, and never using it internally. It's just got too many limitations, as we're seeing here.
On import, we check for valid leaders, and warn if not.
On export, we calculate the leader, and if it exceeds 99999, we warn. We can develop several options for dealing with this, like truncating to the appropriate length, skipping the record or just exporting with an invalid leader.
Cheers,
-Ian
On Tue, Apr 10, 2012 at 11:05, Julian Maurice < julian.maurice@biblibre.com> wrote:
On 10/04/2012 16:27, Jared Camins-Esakov wrote:
Julian,
I know this is an old discussion but I just realized that saving a biblio record longer than 99999 bytes result in a malformed leader. Length of the record takes more than 5 characters and therefore remaining characters in the leader are shifted to the right. This cause the leader to have more than 24 characters.
I'm not sure of what have to be done. Should MARC::Record die when encountering such a record? or should it force length to be 99999? or should the record be truncated to fit the spec? Has someone ever faced this problem? How did you handle it?
MARC::Record should proceed blithely on, like Admiral Farragut at the Battle of Mobile Bay or the Light Brigade at the Battle of Balaclava. The record can still be serialized as MARCXML, and large records are not uncommon in the wild. As far as I know, the only way to handle it is to use MARCXML instead of iso2709.
Regards, Jared
Ok, but the malformed leader is saved in the MARCXML, so it's displayed in the MARC editor, and cataloguing plugins marc21_leader.pl and unimarc_leader.pl won't work correctly. Isn't there any workaround for this?
More than that, Koha is still generating malformed MARC records, and I think it's a problem for an ILS which is MARC-compliant.
-- Julian Maurice <julian.maurice@biblibre.com> BibLibre ______________________________**_________________ Koha-devel mailing list Koha-devel@lists.koha-**community.org<Koha-devel@lists.koha-community.org> http://lists.koha-community.**org/cgi-bin/mailman/listinfo/**koha-devel<http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.**org/<http://bugs.koha-community.org/>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
<fridolyn.somers@gmail.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Op 11-04-12 23:12, Ian Walls schreef:
Is the record length portion of the leader really useful to anyone anymore? It's really a much more valuable concept 30 years ago, when storage in bytes was a real consideration.
It's not really for that, it's to allow seeking through records on tape to be efficient (at least, the way it's designed suggests that to me.) This also shouldn't be a consideration in this day and age.
I'd say just stop counting at 99999, and let the export scripts provide the recalculation, truncation or warning. Since we don't include items internally in our MARC except for indexing, we shouldn't run into this situation very often now.
This would be OK for a single record. My concern is if we ever end up with a block of USMARC records glommed together, could we separate them out without a correct length header? Is there an "end of record" flag?
Novel idea: what about allowing hexidecimal values for those 5 characters? That'd give us an extra 948577 bytes, which would cover most cases, I'd think.
I think that's a bandaid on top of something that needs a real solution, like avoiding USMARC. Technically the length part in MARCXML is unnecessary, I expect that it is just there so that you can simply and procedurally convert between the two formats. Also, it'll create a terrible nightmare when interacting with other systems that get upset by incorrect length headers. I think we'd be better off failing in a graceful manner/working around by setting to 99999, and avoiding the use of USMARC as much as possible. Maybe we could then move a step past that and avoid the use of MARC entirely ;) I heard RDA was the new hotness... Robin.
2012/4/11 Robin Sheat <robin@catalyst.net.nz>
Op 11-04-12 23:12, Ian Walls schreef:
I'd say just stop counting at 99999, and let the export scripts provide the recalculation, truncation or warning. Since we don't include items internally in our MARC except for indexing, we shouldn't run into this situation very often now.
This would be OK for a single record. My concern is if we ever end up with a block of USMARC records glommed together, could we separate them out without a correct length header? Is there an "end of record" flag?
Even if there is not an "EOR" flag, could not the first char of the next header allow quick deduction of the EOR? Kind Regards, Chris
Op 11-04-12 23:12, Ian Walls schreef:
It's not really for that, it's to allow seeking through records on tape to be efficient (at least, the way it's designed suggests that to me.) I'm old enough to confirm thats the purpose although even back in those days you kept encountering incorrect values it was always easier to kludge the reader software than drum the concept of standards into record suppliers heads,
This would be OK for a single record. My concern is if we ever end up with a block of USMARC records glommed together, could we separate them out without a correct length header? Is there an "end of record" flag? Each record ends with a hex 1D Record Terminator character. Best MARC handling library I've used in the past encoded the leader internally onto its own fields, holding the record length as an integer,
On Wed, Apr 11, 2012 at 11:34:07PM +1200, Robin Sheat wrote: then converting that on output. Apart from a couple of characters flagging the bib level and type the bulk of the leader is noise to most users and should be the softwares responsibilty to maintain correctly. As long as you can generate valid MARC on output and feed it in how you hold the record internally is flexible. Colin -- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2 http://www.ptfs-europe.com
participants (12)
-
Bob Birchall -
Chris Nighswonger -
Colin Campbell -
Fridolyn SOMERS -
Galen Charlton -
Ian Walls -
Jared Camins-Esakov -
Julian Maurice -
Mike Hafen -
MJ Ray -
Partha Mukhopadhyay -
Robin Sheat