RFC 3.2: stop copying item data to MARC and MARCXML
Hi, Currently, item record data exists both in the items table and as MARC fields in the bib record (in biblioitems.marc and biblioitems.marcxml). The MARC serialization of the item record is controlled by the MARC framework, and unless one wants a maintenance headache, the same mapping of item fields to MARC subfields must be used for all frameworks. This has several deleterious effects: * updating an item record (e.g., in circulation), forces the ISO 2709 and MARCXML versions of the bib record to be recreated. This is an expensive operation, and if a bib record has several hundred items a checkout or return can take several seconds (or longer) to complete. * If a bib has a large number of items, the ISO 2709 version can go over the maximum length - see bug 2453. * unless one makes a special effort, one is stuck with the same MARC item mapping for both import and export. Some libraries need to import bib records from more than one source and don't always have the ability to dictate that their Koha item field mapping be used by the record supplier. * maintaining two versions of the same data can and has led to bugs. * changing the item field mapping requires one to change Zebra (and NoZebra) index definitions. * this is a denormalization of the database schema, and one that doesn't provide any performance gains. Consequently, I propose the following changes, in order of priority: 1. Take item data out of biblioitems.marc*. Instead, whenever item records need to be embedded in a MARC bib record (primarily during Zebra indexing or record export), do it on the fly. If a bib record has lots of items, this will still be an expensive operation, but it will take place in a batch job or indexing daemon. 2. Separate the item field mapping from the rest of the MARC framework. Allow multiple item field mappings, each identified by a code, and add the ability to specify a mapping to be used during record import, export, or indexing. 3. Switch to Zebra's DOM filter and supply item data for indexing outside of the MARC bib record; instead, define an XML schema to wrap bib, item, and summary records without force the last two to be embedded in the MARC. This is similar to an idea that I believe Tümer Garip proposed back in 2006. Regards, Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt
Galen Charlton a écrit :
Consequently, I propose the following changes, in order of priority:
1. Take item data out of biblioitems.marc*. Instead, whenever item records need to be embedded in a MARC bib record (primarily during Zebra indexing or record export), do it on the fly. If a bib record has lots of items, this will still be an expensive operation, but it will take place in a batch job or indexing daemon. 2. Separate the item field mapping from the rest of the MARC framework. Allow multiple item field mappings, each identified by a code, and add the ability to specify a mapping to be used during record import, export, or indexing. 3. Switch to Zebra's DOM filter and supply item data for indexing outside of the MARC bib record; instead, define an XML schema to wrap bib, item, and summary records without force the last two to be embedded in the MARC. This is similar to an idea that I believe Tümer Garip proposed back in 2006.
this is a good idea. BUT : it means making heavy changes in Koha, as it's the core of most features we have. Even if we have nice packages & a stable API, I think the changes will be heavy. I spoke of that with hdl this morning, he fully agree. So I think this change must be carefully estimated, and double checked if we can include that in 3.2 with the timeline you've announced (and that I fully agree with) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc NOUVEAU TELEPHONE : 04 91 81 35 08
Yes, this will be a significant change, but one I think is necessary. It will only be more difficult to test later when more features are added. Serial records become unmanageable when they On Tue, Sep 30, 2008 at 8:26 AM, Paul POULAIN <paul.poulain@free.fr> wrote:
Galen Charlton a écrit :
Consequently, I propose the following changes, in order of priority:
1. Take item data out of biblioitems.marc*. Instead, whenever item records need to be embedded in a MARC bib record (primarily during Zebra indexing or record export), do it on the fly. If a bib record has lots of items, this will still be an expensive operation, but it will take place in a batch job or indexing daemon. 2. Separate the item field mapping from the rest of the MARC framework. Allow multiple item field mappings, each identified by a code, and add the ability to specify a mapping to be used during record import, export, or indexing. 3. Switch to Zebra's DOM filter and supply item data for indexing outside of the MARC bib record; instead, define an XML schema to wrap bib, item, and summary records without force the last two to be embedded in the MARC. This is similar to an idea that I believe Tümer Garip proposed back in 2006.
this is a good idea. BUT : it means making heavy changes in Koha, as it's the core of most features we have. Even if we have nice packages & a stable API, I think the changes will be heavy. I spoke of that with hdl this morning, he fully agree.
So I think this change must be carefully estimated, and double checked if we can include that in 3.2 with the timeline you've announced (and that I fully agree with)
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc NOUVEAU TELEPHONE : 04 91 81 35 08 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Ryan Higgins LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704
Oops -- hit send instead of save :/ to complete my thought : On Tue, Sep 30, 2008 at 1:36 PM, Ryan Higgins <ryan.higgins@liblime.com>wrote:
Yes, this will be a significant change, but one I think is necessary. It will only be more difficult to test later when more features are added.
Serial records become unmanageable when they
... reach a couple hundred items. We had a couple of bugs caused by the duplication of the data in early 3.0 that were subsequently fixed, but caused the data in marc vs items table to become out of sync. I think this move will improve performance quite a bit, as it not only affects many-item records, but our availability and circ status. Some quick profiling reveals that xml parsing is by far the greatest component of most operations in Koha. Removing the need to parse the full bib record on every circ transaction should be a good step in improving performance imo. Ryan On Tue, Sep 30, 2008 at 8:26 AM, Paul POULAIN <paul.poulain@free.fr> wrote:
Galen Charlton a écrit :
Consequently, I propose the following changes, in order of priority:
1. Take item data out of biblioitems.marc*. Instead, whenever item records need to be embedded in a MARC bib record (primarily during Zebra indexing or record export), do it on the fly. If a bib record has lots of items, this will still be an expensive operation, but it will take place in a batch job or indexing daemon. 2. Separate the item field mapping from the rest of the MARC framework. Allow multiple item field mappings, each identified by a code, and add the ability to specify a mapping to be used during record import, export, or indexing. 3. Switch to Zebra's DOM filter and supply item data for indexing outside of the MARC bib record; instead, define an XML schema to wrap bib, item, and summary records without force the last two to be embedded in the MARC. This is similar to an idea that I believe Tümer Garip proposed back in 2006.
this is a good idea. BUT : it means making heavy changes in Koha, as it's the core of most features we have. Even if we have nice packages & a stable API, I think the changes will be heavy. I spoke of that with hdl this morning, he fully agree.
So I think this change must be carefully estimated, and double checked if we can include that in 3.2 with the timeline you've announced (and that I fully agree with)
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc NOUVEAU TELEPHONE : 04 91 81 35 08 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
--
Ryan Higgins
LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704
-- Ryan Higgins LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704
Ryan Higgins a écrit :
I think this move will improve performance quite a bit, as it not only affects many-item records, but our availability and circ status. Some quick profiling reveals that xml parsing is by far the greatest component of most operations in Koha. Removing the need to parse the full bib record on every circ transaction should be a good step in improving performance imo.
but, on every circ, we want to handle the item update in zebra too right ? so we will have to : 1- update the item in SQL 2- rebuild the MARC record with biblio+items 3- reindex zebra with both so won't the gain be very small (gain in step 1, but loss in step 2, that is fast with the actual structure) ? -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc NOUVEAU TELEPHONE : 04 91 81 35 08
Hi, On Wed, Oct 1, 2008 at 8:33 AM, Paul POULAIN <paul.poulain@free.fr> wrote:
but, on every circ, we want to handle the item update in zebra too right ? so we will have to : 1- update the item in SQL 2- rebuild the MARC record with biblio+items 3- reindex zebra with both
so won't the gain be very small (gain in step 1, but loss in step 2, that is fast with the actual structure) ?
Since 2 and 3 will be happening in a separate process (and could be configured to happen on a separate server), the enhancement to transaction speed in the OPAC and staff interface will be significant. Furthermore, it would open up possibilities for changing how we do indexing with Zebra so that we don't need to recreate the entire bib + holdings + items XML every time an item is updated. Regards, Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt
Galen Charlton a écrit :
Hi,
On Wed, Oct 1, 2008 at 8:33 AM, Paul POULAIN <paul.poulain@free.fr> wrote:
but, on every circ, we want to handle the item update in zebra too right ? so we will have to : 1- update the item in SQL 2- rebuild the MARC record with biblio+items 3- reindex zebra with both
so won't the gain be very small (gain in step 1, but loss in step 2, that is fast with the actual structure) ?
Since 2 and 3 will be happening in a separate process (and could be configured to happen on a separate server), the enhancement to transaction speed in the OPAC and staff interface will be significant. Furthermore, it would open up possibilities for changing how we do indexing with Zebra so that we don't need to recreate the entire bib + holdings + items XML every time an item is updated.
can we have biblios & items in 2 separate DB that can be z3950/SRU-SRW queried in only 1 query (to get the biblio & the items, like what we have atm) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc NOUVEAU TELEPHONE : 04 91 81 35 08
participants (3)
-
Galen Charlton -
Paul POULAIN -
Ryan Higgins