New KOHA3 API -design issue
Hi all, If Joshua approves you will soon be seeing a big commit to head that changes the API of KOHA 3 very soon. Here is what I have done so that we can share comments and improvements on this. Major changes: 1-Bibliographic, Holdings and Authority records are all separate MARC records. 2-All searches rely on ZEBRA indexing and searching (Z3950 compliant) 3-Non MARC data handling is dropped. Non-marc like display is kept 4-All data is UTF-8. No other encoding supported as this support all languages 5-Change affects all data handling so for a short time breaks all modules (circulation, serials,acquistions and the lot) 6-Templates not affected More detail: Database design: 1-biblioitems, marc_words,stopwords, marc_subfield_table, autth_subield_table, additionalauthors, subjects,categories, marc_blob tables completely dropped 2- biblio table now has: biblionumber,marc,itemtype,framework 3-items table now has : itemnumber,biblionumber,barcode,marc 4- auth_header table now has: authid,marc 5- 1 new frameworks table added similar to marc frameworks for holdings marc 6- A new koha_attr table added. This one is for creating virtual fields which we do not have any more. Also helps matching your own MARC indexing rules to KOHA and ZEBRA. (More on this later) Design issues; 1- itemtype definition still kept at bibliographic level. Can be very easily converted to item level. i.e one bibliographic record with different itemtypes (book,cd,dvd etc).Open for discussion 2- All items (holdings) are single marc records, instead of all items in one marc record for the same bibliographic record. The main idea behind this is flexibility and speed. One full record for an item gives lots of fields to put all that information that you wanted to keep but did not have extra subfields left. All data including circulation data is kept within the marc records, this approach makes circulation updates faster as we do not have to reindex unchanged data related to other items again (zebra technicalities!). 3- All MARC framework designs have OPAC:show-hide Intranet:show-hide Editor:show-show collapsed-hide as dropdown selections so not to wory about what we put in these MARC fields. All handled correctly 4- Circulation speed is improved. One marc update instead of 6 sepatrate database update calls we used to have. Currently working modules: -MARC editor works - Item screen although looks the same now has multiple fields and works - authories adding editing works but needs to be improved to bibliographic editor design -Circulation works but being updated for faster handling -Serials and acquisitions needs work The new koha_attr table: We used to match our marc frameworks to table field names for both being multi-marc compliant and for template design issues. These field names now do not exist. So now we create virtula ones by a user editable screen. Although it comes filled with all the previous field names (and many more extras) to keep all koha modules and templates working you can add as many as you want when the need arise. No need to change database structures. But a table just for this feature was not needed, we could have had those names as authorised values so i added another functionality to this table. It matches your KOHA field names to ZEBRA indexing. This way the majority of koha code stays intact and field names joing our records like subcrisptionid itemnumber barcode etc can function as previous and removes the strain on programmers to either programme with PQF,CQL or CCL. A routine supplied converts everything to zebra language. This approach also release the strain on developers on trying to match what zebra is indexing and what koha thinks or should be indexed. Specially at this development stage until a KOHA minimum indexing standarts is defined this gives both the programmer and the library freedom of movement. A library may also decide to use completely inhouse designed indexing attributes and not comply with standarts like Bib-1 or Gils. All you have to do is fill you koha attr tables in sysadmin and tell koha what barcode is and what title is.Koha will continue to function. In simpler terms it does what frameworks does but compliments that table and bring ZEBRA in. I will be writing more on this issue after the commits but in the mean time expect to have your comments on all the issues regarding this upgrade. Cheers Tumer Garip NEU Grand Library
Tümer Garip a écrit :
Hi all,
Hi Tümer,
2- biblio table now has: biblionumber,marc,itemtype,framework 3-items table now has : itemnumber,biblionumber,barcode,marc
I'm *strongly against* dropping all the values. Joshua and me have spoken of this before, and I really think we MUST kept a minimum of datas in biblios & items tables. Like title, author, and all values that already exists. It won't cost a lot of disk or CPU, and will be very very useful for at least 2 things : * debuging Koha * querying the database with pure SQL. The other question being also to have iso2709 and (or ?) marcxml in the datas. xml is human readable & turned toward future, where iso is more efficient in term of CPU & disk usage.
6- A new koha_attr table added. This one is for creating virtual fields which we do not have any more. Also helps matching your own MARC indexing rules to KOHA and ZEBRA. (More on this later)
i'm waiting for the "more on this later", because I don't understand what you mean here...
2- All items (holdings) are single marc records, instead of all items in one marc record for the same bibliographic record. The main idea behind this is flexibility and speed. One full record for an item gives lots of fields to put all that information that you wanted to keep but did not have extra subfields left. All data including circulation data is kept within the marc records, this approach makes circulation updates faster as we do not have to reindex unchanged data related to other items again (zebra technicalities!).
sounds correct & good idea to me.
3- All MARC framework designs have OPAC:show-hide Intranet:show-hide Editor:show-show collapsed-hide as dropdown selections so not to wory about what we put in these MARC fields. All handled correctly 4- Circulation speed is improved. One marc update instead of 6 sepatrate database update calls we used to have.
The new koha_attr table: <snip> I will be writing more on this issue after the commits but in the mean time expect to have your comments on all the issues regarding this upgrade. I still don't understand what it's done for, sorry...
-- Paul POULAIN Consultant indépendant en logiciels libres responsable francophone de koha (SIGB libre http://www.koha-fr.org)
Hi Paul, As I work more on this issue of ZEBRA API more problems arise and thats why I have written this document. To have more details in biblio and items tables i also think is not an issue of CPU or disk space. Both approach is still acceptable. I kept the barcode as well as you noticed. My idea is that we mainly use ZEBRA for searching but use SQL to reach a record with unique indexes. This way we can do some load balancing and let zebra do the heavy work. SQL is as fast when reaching unique records. Currently I have a bigger problem that our current zebra design does not allow merged resultsets. I am working heavily on this to see what can be done without too much CPU overhead. i.e. Currently I cannot search a title in a specific library branch. Regarding koha_attr table. Well i needed the old field names to exist somewhere so i can use them in our framework (marc_subfield_structure) designs to map 245$a to "title" 300$a "publisherid" etc. The old API got these names from the tables themselves (biblio,biblioitems,items,subject,additionalauthors). So I created a table with all the old field names in so that all the old templates and most of the code can work with <!--TMPL VAR NAME="title"--> notations. But to create a table just for this was unneccessary so I added extra functionality to it. It has another column called "attr". There we can define what is the corresponding ZEBRA attribute to this virtual_field_name "title", "publisherid", "barcode" or whatever. It has data like this: Koha field ATTR LABEl SORT SERVER itemnumber @attr 1=1032 Item number No itemserver classification @attr 1=16 LC classification No biblioserver conference_name @attr 1=3 Conference name No biblioserver copyrightdate @attr 1=30 @attr 4=5 Copyright date No biblioserver This table now provides me two more functionalities. (I am talking about the backend workings by the way not front end searching). While we evolve ZEBRA functionality we are all using different search attributes for our fields. Such as you are using Identifier-standard (@attr 1=1007) for you biblionumber I am using Number-db (@attr 1=52). Even at one stage we all use the same attributes for indexing in time people will want to add more or change their indexing attributes. This table is Sysytme admin editable. So within KOHA API I still call biblionumber=some number and do not use any @attr notation at all. The code is free of Bib-1 or GILS or KOHA designed attributes. The second functionality is that i can now create as many virtual fields as i want on the fly. As you may have noticed I have conference_name defined. So if I index my zebra with conference_name as long as define it in my koha_attr it will appear as a searchable field in my search screens. All these field names do not have to be mapped to marc_subfield_structures but if you want to use them in your templates as <!--TMPL VAR NAME="conference_name"--> than you have to map tham to your MARC tables ofcourse. One other functionality is that we do not have to hardcode $9 where authority id is going to reside in our MARC definitions. It has a name now "authid". You just map this to any record any field and dont bother whether its $9 $6 or $8 you are using your link from. So flexible linking is possible. Similar to MARCfind_marc_from_kohafield we now have MARCfind_attr_from_kohafield. And lastly it says whether you sorted on that field or not. If you did then it gets included in your sort dropdown menus. No hardcoding. You may decide to sort on a field that you do not think you needed now. When you do just set the flag sorts to true and your Search page will have it included. Whether we use PQF,CQL,CCL in the frontend for searching purposes (which I belive we should be able to do them all) in the background I needed something flexible so that developers can go on programming without having to concern of whats happening with atributes etc. I hope this explains it better and I await your comments on this. Cheers Tumer -----Original Message----- From: Paul POULAIN [mailto:paul.poulain@free.fr] Sent: Monday, July 17, 2006 6:21 PM To: Tümer Garip Cc: koha-devel@nongnu.org Subject: Re: [Koha-devel] New KOHA3 API -design issue Tümer Garip a écrit :
Hi all,
Hi Tümer,
2- biblio table now has: biblionumber,marc,itemtype,framework 3-items table now has : itemnumber,biblionumber,barcode,marc
I'm *strongly against* dropping all the values. Joshua and me have spoken of this before, and I really think we MUST kept a minimum of datas in biblios & items tables. Like title, author, and all values that already exists. It won't cost a lot of disk or CPU, and will be very very useful for at least 2 things : * debuging Koha * querying the database with pure SQL. The other question being also to have iso2709 and (or ?) marcxml in the datas. xml is human readable & turned toward future, where iso is more efficient in term of CPU & disk usage.
6- A new koha_attr table added. This one is for creating virtual fields which we do not have any more. Also helps matching your own MARC indexing rules to KOHA and ZEBRA. (More on this later)
i'm waiting for the "more on this later", because I don't understand what you mean here...
2- All items (holdings) are single marc records, instead of all items in one marc record for the same bibliographic record. The main idea behind this is flexibility and speed. One full record for an item gives lots of fields to put all that information that you wanted to keep but did not have extra subfields left. All data including circulation data is kept within the marc records, this approach makes circulation updates faster as we do not have to reindex unchanged data
related to other items again (zebra technicalities!).
sounds correct & good idea to me.
3- All MARC framework designs have OPAC:show-hide Intranet:show-hide Editor:show-show collapsed-hide as dropdown selections so not to wory about what we put in these MARC fields. All handled correctly 4- Circulation speed is improved. One marc update instead of 6 sepatrate database update calls we used to have.
The new koha_attr table: <snip> I will be writing more on this issue after the commits but in the mean
time expect to have your comments on all the issues regarding this upgrade. I still don't understand what it's done for, sorry...
-- Paul POULAIN Consultant indépendant en logiciels libres responsable francophone de koha (SIGB libre http://www.koha-fr.org)
Tümer Garip a écrit :
Hi Paul,
As I work more on this issue of ZEBRA API more problems arise and thats why I have written this document. To have more details in biblio and items tables i also think is not an issue of CPU or disk space. Both approach is still acceptable. I kept the barcode as well as you noticed. My idea is that we mainly use ZEBRA for searching but use SQL to reach a record with unique indexes. This way we can do some load balancing and let zebra do the heavy work. SQL is as fast when reaching unique records.
Currently I have a bigger problem that our current zebra design does not allow merged resultsets. I am working heavily on this to see what can be done without too much CPU overhead. i.e. Currently I cannot search a title in a specific library branch.
the problem is a consequence to the splitting of items & biblios in 2 different zebra DB isn't it ? However, thx, it's much much much more clear to me. and i'm ++ FOR this solution. The need for some semantically easy to understand datas in SQL is stats & debugging & ... purposes. So I think we need only a few values in biblio tables. Something like title/author/itemtype. All other values could be "moved to" koha_attr new table.
The second functionality is that i can now create as many virtual fields as i want on the fly. As you may have noticed I have conference_name defined. So if I index my zebra with conference_name as long as define it in my koha_attr it will appear as a searchable field in my search screens. All these field names do not have to be mapped to marc_subfield_structures but if you want to use them in your templates as <!--TMPL VAR NAME="conference_name"--> than you have to map tham to your MARC tables ofcourse.
except that any virtual field would have to be added to zebra config file as well, needing a reindexing ? or am I missing something ?
And lastly it says whether you sorted on that field or not. If you did then it gets included in your sort dropdown menus. No hardcoding. You may decide to sort on a field that you do not think you needed now. When you do just set the flag sorts to true and your Search page will have it included.
iiuc, the advanced search feature could also take benefits of this koha_attr table : the library could define which "fields" they want to have in the list. That's exactly something requested yesterday !!! (in koha_attr, we could have a "field for opac" checkbox : if unchecked, the search is only for librarians, otherwise, it's for both interfaces)
I hope this explains it better and I await your comments on this.
you've got them, read you later. -- Paul POULAIN Consultant indépendant en logiciels libres responsable francophone de koha (SIGB libre http://www.koha-fr.org)
On Mon, July 17, 2006 7:53 pm, Tümer Garip wrote:
It has another column called "attr". There we can define what is the corresponding ZEBRA attribute to this virtual_field_name "title", "publisherid", "barcode" or whatever.
The appropriate table or other configuration location would seem to need more than merely the designation of an attribute or attribute sequence from the Bib-1 attribute set corresponding to the original Koha SQL database columns if the goal is to avoid hard coding in Perl. Sufficient flexible generalisation of the Perl code would need at least one more column in addition to a column for attributes. There ought to be a column for the attribute set, without presuming Bib-1 is the attribute set for a given attribute sequence appearing in koha_attr.attr. We might presume Bib-1 to be the default attribute set for some basic context but we know it may not be the exclusive attribute set. Bib-1 is not sufficient for every purpose. Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com 212-674-3783 On Mon, July 17, 2006 7:53 pm, Tümer Garip wrote: [snip]
Regarding koha_attr table. Well i needed the old field names to exist somewhere so i can use them in our framework (marc_subfield_structure) designs to map 245$a to "title" 300$a "publisherid" etc. The old API got these names from the tables themselves (biblio,biblioitems,items,subject,additionalauthors). So I created a table with all the old field names in so that all the old templates and most of the code can work with <!--TMPL VAR NAME="title"--> notations. But to create a table just for this was unneccessary so I added extra functionality to it. It has another column called "attr". There we can define what is the corresponding ZEBRA attribute to this virtual_field_name "title", "publisherid", "barcode" or whatever. It has data like this: Koha field ATTR LABEl SORT SERVER itemnumber @attr 1=1032 Item number No itemserver classification @attr 1=16 LC classification No biblioserver conference_name @attr 1=3 Conference name No biblioserver copyrightdate @attr 1=30 @attr 4=5 Copyright date No biblioserver
This table now provides me two more functionalities. (I am talking about the backend workings by the way not front end searching). While we evolve ZEBRA functionality we are all using different search attributes for our fields. Such as you are using Identifier-standard (@attr 1=1007) for you biblionumber I am using Number-db (@attr 1=52). Even at one stage we all use the same attributes for indexing in time people will want to add more or change their indexing attributes. This table is Sysytme admin editable. So within KOHA API I still call biblionumber=some number and do not use any @attr notation at all. The code is free of Bib-1 or GILS or KOHA designed attributes.
[snip]
participants (3)
-
Paul POULAIN -
Thomas Dukleth -
Tümer Garip