RFC: Koha : Merge biblio & biblioitems tables
Late entry, I know, but I would like to get rid of the clunky extra table, and merge the columns that we need from biblioitems into a new biblio table. I've listed the columns to keep on the wiki; please review and add any that are missing. http://wiki.koha.org/doku.php?id=en:development:rfcs3.2:rfc32_biblio_table -- Ryan Higgins LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704
Ryan Higgins a écrit :
Late entry, I know, but I would like to get rid of the clunky extra table, and merge the columns that we need from biblioitems into a new biblio table. I've listed the columns to keep on the wiki; please review and add any that are missing.
I totally agree with this idea. As we are doing this, could we change the table name (biblios). I think it should be written in our coding guidelines that table names are plural. -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc NOUVEAU TELEPHONE : 04 91 81 35 08
Paul POULAIN wrote:
As we are doing this, could we change the table name (biblios). I think it should be written in our coding guidelines that table names are plural.
Nitpick: if all tables end in "s", isn't the "s" redundant? I always name tables sans plural. When you ask for a record from the "biblio" table using ORM, you get a Biblio object, not a Biblios object. When you ask DBI to SELECT a row from the biblio table, you get a biblio hash. (Yes, of course you can ask for many in the SELECT, but then you get an array of biblio). Since we are going to proceed to using ORM, I would not want to see the plural used on the data model classes, i.e. an instance of C4::Borrowers implies it is a class that has *many* borrowers in an instance. This is inaccurate. A C4::Borrower object represents a single borrower. cheers rickw -- _________________________________ Rick Welykochy || Praxis Services Tis the dream of each programmer before his life is done, To write three lines of APL and make the damn thing run.
Rick Welykochy a écrit :
Paul POULAIN wrote:
As we are doing this, could we change the table name (biblios). I think it should be written in our coding guidelines that table names are plural.
Nitpick: if all tables end in "s", isn't the "s" redundant?
In fact I don't care. But atm we have some tables plural, and some singular. and for sure, that's a pity ! I'm for a unique rule ! and there are more plural than singular, so that would be easier to switch to plural I think. -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc NOUVEAU TELEPHONE : 04 91 81 35 08
Hi, On Tue, Nov 25, 2008 at 4:24 AM, Paul POULAIN <paul.poulain@free.fr> wrote:
In fact I don't care. But atm we have some tables plural, and some singular. and for sure, that's a pity ! I'm for a unique rule ! and there are more plural than singular, so that would be easier to switch to plural I think.
Roughly speaking, 70% of the table names have plural names and 30% singular. Although it would be nice to pick a convention and change all table names to match, I'm not in favor of renaming biblio to biblios at this point due to the number of queries on that table that would have to be changed and the risk of adding bugs. We'll be better able to make such name changes once an ORM is in place. Regards, Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt
Or we have the option of not caring at all once an ORM is in place! --joe On Tue, Nov 25, 2008 at 7:23 AM, Galen Charlton <galen.charlton@liblime.com>wrote:
Hi,
On Tue, Nov 25, 2008 at 4:24 AM, Paul POULAIN <paul.poulain@free.fr> wrote:
In fact I don't care. But atm we have some tables plural, and some singular. and for sure, that's a pity ! I'm for a unique rule ! and there are more plural than singular, so that would be easier to switch to plural I think.
Roughly speaking, 70% of the table names have plural names and 30% singular. Although it would be nice to pick a convention and change all table names to match, I'm not in favor of renaming biblio to biblios at this point due to the number of queries on that table that would have to be changed and the risk of adding bugs. We'll be better able to make such name changes once an ORM is in place.
Regards,
Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Joe, are you suggesting that merging the tables will not be worth the effort? I guess it really isn't going much of a performance hit to add an extra table join to queries since it's 1-1 and indexed. Perhaps the time spent on this would be better spent on ORM ? The only benefit that this has if it's behind an ORM is potentially making direct sql reporting easier and peace of mind that the table structure is more sane than it used to be. So is the performance difference negligible in keeping two tables ? If so, perhaps just removing the extra columns and fixing duplicated column names like 'notes' would be sufficient, and we can put off the merge until after the ORM is in place. My reasoning for doing the merge first was to clean up before I had to touch some of these queries for some work I'm looking at that won't wait until after the ORM changes are in place. But I'd rather put time in where it's most valuable short-term. Comments ? Ryan On Tue, Nov 25, 2008 at 12:43 PM, Joe Atzberger <ohiocore@gmail.com> wrote:
Or we have the option of not caring at all once an ORM is in place! --joe
On Tue, Nov 25, 2008 at 7:23 AM, Galen Charlton < galen.charlton@liblime.com> wrote:
Hi,
On Tue, Nov 25, 2008 at 4:24 AM, Paul POULAIN <paul.poulain@free.fr> wrote:
In fact I don't care. But atm we have some tables plural, and some singular. and for sure, that's a pity ! I'm for a unique rule ! and there are more plural than singular, so that would be easier to switch to plural I think.
Roughly speaking, 70% of the table names have plural names and 30% singular. Although it would be nice to pick a convention and change all table names to match, I'm not in favor of renaming biblio to biblios at this point due to the number of queries on that table that would have to be changed and the risk of adding bugs. We'll be better able to make such name changes once an ORM is in place.
Regards,
Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Ryan Higgins LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704
No, Ryan, I was just talking about the singular vs. plural table names. As in, nobody will complain about the table names when they are taken out of view. I'm still very much for killing biblioitems. On Tue, Nov 25, 2008 at 1:44 PM, Ryan Higgins <ryan.higgins@liblime.com>wrote:
Joe, are you suggesting that merging the tables will not be worth the effort? I guess it really isn't going much of a performance hit to add an extra table join to queries since it's 1-1 and indexed. Perhaps the time spent on this
would be better spent on ORM ? The only benefit that this has if it's behind an ORM is potentially making direct sql reporting easier and peace of mind that the table structure is more sane than it used to be.
So is the performance difference negligible in keeping two tables ?
If so, perhaps just removing the extra columns and fixing duplicated column names like 'notes' would be sufficient, and we can put off the merge until after the ORM is in place.
My reasoning for doing the merge first was to clean up before I had to touch some of these queries for some work I'm looking at that won't wait until after the ORM changes are in place.
But I'd rather put time in where it's most valuable short-term.
Comments ?
Ryan
On Tue, Nov 25, 2008 at 12:43 PM, Joe Atzberger <ohiocore@gmail.com>wrote:
Or we have the option of not caring at all once an ORM is in place! --joe
On Tue, Nov 25, 2008 at 7:23 AM, Galen Charlton < galen.charlton@liblime.com> wrote:
Hi,
On Tue, Nov 25, 2008 at 4:24 AM, Paul POULAIN <paul.poulain@free.fr> wrote:
In fact I don't care. But atm we have some tables plural, and some singular. and for sure, that's a pity ! I'm for a unique rule ! and there are more plural than singular, so that would be easier to switch to plural I think.
Roughly speaking, 70% of the table names have plural names and 30% singular. Although it would be nice to pick a convention and change all table names to match, I'm not in favor of renaming biblio to biblios at this point due to the number of queries on that table that would have to be changed and the risk of adding bugs. We'll be better able to make such name changes once an ORM is in place.
Regards,
Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Ryan Higgins
LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704
Hi, On Tue, Nov 25, 2008 at 2:02 PM, Joe Atzberger <ohiocore@gmail.com> wrote:
No, Ryan, I was just talking about the singular vs. plural table names. As in, nobody will complain about the table names when they are taken out of view. I'm still very much for killing biblioitems.
On the other hand, an ORM would not eliminate the need for a user to write SQL for ad hoc reports, so there is still a case to be made for making the table names consistent. It just becomes easier to do that once the ORM is fully employed. Regards, Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt
Ryan Higgins wrote:
So is the performance difference negligible in keeping two tables ?
Who really cares about machine performance these days. CPU, memory and storage can be doubled at the drop of a hat. Why are we planning to put in an ORM? Not to speed up the code. To increase people efficiency. <RANT> I am *VERY* concerned about people performance. To wit: (*) the number of bugs being uncovered in Koha increases monotonically day by day; they are not being fixed fast enough (*) it is very difficult to maintain and fix bugs in some of the areas of the Koha software; this results in a people inefficiency, i.e. it can take hours to find one little shard of error in the messy Perl 4.0 coding we have to live with (*) the "organic" way that Koha has grown in leaps and bounds over the years with no overall analysis and design has meant that once again people (i.e. developer) efficiency suffers (*) the lack of peer review of existing code in Koha; there are many nasty and out-dated coding styles lurking therein if you care to look; (*) the lack of audit of database access; i.e. when I walk through the code, sometimes I can see that the same bits of data are loaded multiple times form the database, or worse, ten rounds trips are used to the database when one would do (piggy backing) (*) software developers start to shy away and eventually shun bady designed and written systems since it becomes a nightmare to care for them and keep them up and running reliably. If you don't know what I am talking about, perhaps a new Wiki page is required to explain people vs machine efficiency. Those who have been around the block a few times will read the above as the same old repetitive litany of many software projects that need care and a helping hand. It is very easy for geeks to comment on how slow this piece of Perl code is or how inefficient that line of SQL is, but rarely do we address the real solid underlying issues which involve managing people within the project and helping them to work efficiently. Let's get some perspective on Koha development: the problems that Koha suffer ain't gonna go away if we improve the efficiency of the code and the data model. They will only be addressed and resolved if we manage the people resources we have available to fix things. And especially only if it becomes easier to understand the Koha code base and more efficient to manage and change it. </RANT> cheers rickw -- _________________________________ Rick Welykochy || Praxis Services Tis the dream of each programmer before his life is done, To write three lines of APL and make the damn thing run.
Rick <RANT> What you say may be all well and good, but I have yet to see a single patch from you. Want some bugs fixed? How bout fixing a couple yourself, show us all how its done eh? </RANT> On Wed, Nov 26, 2008 at 12:40 PM, Rick Welykochy <rick@praxis.com.au> wrote:
Ryan Higgins wrote:
So is the performance difference negligible in keeping two tables ?
Who really cares about machine performance these days. CPU, memory and storage can be doubled at the drop of a hat.
Why are we planning to put in an ORM? Not to speed up the code. To increase people efficiency.
<RANT>
I am *VERY* concerned about people performance. To wit:
(*) the number of bugs being uncovered in Koha increases monotonically day by day; they are not being fixed fast enough
(*) it is very difficult to maintain and fix bugs in some of the areas of the Koha software; this results in a people inefficiency, i.e. it can take hours to find one little shard of error in the messy Perl 4.0 coding we have to live with
(*) the "organic" way that Koha has grown in leaps and bounds over the years with no overall analysis and design has meant that once again people (i.e. developer) efficiency suffers
(*) the lack of peer review of existing code in Koha; there are many nasty and out-dated coding styles lurking therein if you care to look;
(*) the lack of audit of database access; i.e. when I walk through the code, sometimes I can see that the same bits of data are loaded multiple times form the database, or worse, ten rounds trips are used to the database when one would do (piggy backing)
(*) software developers start to shy away and eventually shun bady designed and written systems since it becomes a nightmare to care for them and keep them up and running reliably.
If you don't know what I am talking about, perhaps a new Wiki page is required to explain people vs machine efficiency.
Those who have been around the block a few times will read the above as the same old repetitive litany of many software projects that need care and a helping hand.
It is very easy for geeks to comment on how slow this piece of Perl code is or how inefficient that line of SQL is, but rarely do we address the real solid underlying issues which involve managing people within the project and helping them to work efficiently.
Let's get some perspective on Koha development: the problems that Koha suffer ain't gonna go away if we improve the efficiency of the code and the data model.
They will only be addressed and resolved if we manage the people resources we have available to fix things. And especially only if it becomes easier to understand the Koha code base and more efficient to manage and change it.
</RANT>
cheers rickw
-- _________________________________ Rick Welykochy || Praxis Services
Tis the dream of each programmer before his life is done, To write three lines of APL and make the damn thing run. _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Chris Cormack wrote:
<RANT> What you say may be all well and good, but I have yet to see a single patch from you. Want some bugs fixed? How bout fixing a couple yourself, show us all how its done eh? </RANT>
Stay tuned :) Although, I have no plans to rewrite the Koha data model and install an ORM by myself. The more I think about that, the more I see a need for introducing and adhering to some "best practices" for software systems into Koha, i.e. can we please consider the following steps to be taken before we just start writing code, i.e. (*) requirement specs (*) analysis of those to produce some design guidelines (*) design of the new and modified software (*) NOW WE CAN CODE IT UP! (*) testing systems (*) installation, configuration and deployment mgmt There are peer reviews at each of the above steps. And there are more steps I haven't mentioned. Some of the above steps are being done now, in an ad hoc manner. Failing to adhere to best practices in software projects means that the developers suffer at the cost of the quality of the delivered product. I am more than willing to share my experiences and experience in this realm to assist in getting Koha to stand out as a quality FOSS project. Please don't read any criticism I write here as an attack. I am attempting to introduce some quality that is lacking. If you review my posts to this list over the past few years, you will see that I have introduced some changes towards that goal. I am not here to whine and complain. I have much better things to do with my time. cheers rickw -- _________________________________ Rick Welykochy || Praxis Services You go find out what they need and I'll start coding. -- two geeks
Hi, On Tue, Nov 25, 2008 at 7:30 PM, Rick Welykochy <rick@praxis.com.au> wrote:
Although, I have no plans to rewrite the Koha data model and install an ORM by myself.
Good thing that you don't have to do it by yourself, then, and that there are others discussing, planning, and coding this. You've seen Andy's trial patches; please feel free to send supplementary patches or even counterpatches.
The more I think about that, the more I see a need for introducing and adhering to some "best practices" for software systems into Koha, i.e. can we please consider the following steps to be taken before we just start writing code, i.e.
(*) requirement specs
(*) analysis of those to produce some design guidelines
(*) design of the new and modified software
Which is currently being done via the RFC process.
(*) NOW WE CAN CODE IT UP!
(*) testing systems
(*) installation, configuration and deployment mgmt
There are peer reviews at each of the above steps. And there are more steps I haven't mentioned.
Peer review is open to anybody who cares to subscribe to the koha-patches list and comments. Are you volunteering to help with patch review?
Please don't read any criticism I write here as an attack. I am attempting to introduce some quality that is lacking.
Well, good. Nobody is perfect, no software project is perfect, there's always room for improvement, and good ideas can come from anyone. However, talk is cheap: ultimately, the best way to make your case is to fix problems and deliver quality patches that others find worthy of emulation. Don't like gaps in the project's process? Step up and fill one of them. Regards, Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt
On Tue, Nov 25, 2008 at 5:40 PM, Rick Welykochy <rick@praxis.com.au> wrote:
<RANT>
I am *VERY* concerned about people performance. To wit:
Hi Rick - I'm thankful for your rant. Not only do I agree that we're suffering from some technical debt that makes contributions more difficult, time consuming, and laborious, but I see notes like yours as more of a motivational encouragement than a repetition of old arguments. I try to make my new code better than what came before it, and to make some of the surrounding code a little better, too. I hope you do the same and to help me and others improve. Here are some things that have come along recently that I view as helping in this front. I urge you to support them and to help improve the Koha code in other ways. * We have a perltidy rc file at <http://git.koha.org/cgi-bin/gitweb.cgi?p=Koha;a=blob;f=xt/perltidyrc;hb=HEAD> that can help us format our code consistently. Don't forget to run it on your newly contributed files and on the portions of files that you edit. * We have a growing test suite which is run nightly by a cronjob. The reports are sent to a smolder server for easy viewing <http://git.koha.org/cgi-bin/gitweb.cgi?p=Koha;a=blob;f=xt/perltidyrc;hb=HEAD>. We unfortunately often break our own test suite and don't get it fixed very soon, so maintaining that may help. * We have some RFCs for development style improvements or projects to improve how we write code. For instance, it's pretty commonly agreed that we should be turning on 'warnings' in all of the perl code: <http://wiki.koha.org/doku.php?id=en:development:rfcs3.2:rfc32_turn_on_warnings>. And the code that already runs under warnings often emits useless warnings. A project like that is pretty easy to contribute to and helps pay down some of our technical debt. Or, if you're more ambitious, how about working to find a minimal set of Perl::Critic policies that we can agree on and making sure our code is valid by those guidelines? * I drew up some ideas on things to do to help review each others' code: <http://wiki.koha.org/doku.php?id=en:development:code_review_ideas>. Do you have any additional tips there, or suggestions on how to make a process for reviewing each other's code more efficient or valuable? Thanks for pointing out this weakness of the Koha project. I have lots of ideas on how we can improve, too. I'm hopeful that after we pluck some of the lower hanging fruit that has already been described that we can investigate more ways. I think that knowing that most of us are constantly fighting for things to get better helps us to work together towards these improvements. Please don't let your rant stall out before it becomes action! -Andy
On Wed, Nov 26, 2008 at 7:06 AM, Andrew Moore <andrew.moore@liblime.com> wrote:
* We have a growing test suite which is run nightly by a cronjob. The reports are sent to a smolder server for easy viewing <http://git.koha.org/cgi-bin/gitweb.cgi?p=Koha;a=blob;f=xt/perltidyrc;hb=HEAD>. We unfortunately often break our own test suite and don't get it fixed very soon, so maintaining that may help.
oops, bad link: http://arwen.metavore.com:8000/app/public_projects/smoke_reports/1 -A
Andrew Moore a écrit :
Thanks for pointing out this weakness of the Koha project. I have lots of ideas on how we can improve, too.
Can I just add that libraries don't care how it's coded, they want features. So we (at least BibLibre) never are paid to improve code, we are paid to add feature. That should not be an excuse, but that's what we face. However, the positive aspect of this is that, with LibLime & BibLibre growing, we can expect to be able to self-fund the code improvement. -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc NOUVEAU TELEPHONE : 04 91 81 35 08
Andrew Moore wrote:
On Tue, Nov 25, 2008 at 5:40 PM, Rick Welykochy <rick@praxis.com.au> wrote:
<RANT>
I am *VERY* concerned about people performance. To wit:
Hi Rick - I'm thankful for your rant. Not only do I agree that we're suffering from some technical debt that makes contributions more difficult, time consuming, and laborious, but I see notes like yours as more of a motivational encouragement than a repetition of old arguments. I try to make my new code better than what came before it, and to make some of the surrounding code a little better, too. I hope you do the same and to help me and others improve.
[SNIP] Thanks Andrew for taking the time to move my <RANT> forwards to suggestions for some positive action. That is of course what we need here. I am taking onboard all of your recommendations ... one could become *very* busy dipping into all of these pies. I'll see what fits my spare time and "esprit". And I haven't forgotten about your ORM work and suggestions. I just have to get that darn perl modules installed in a clean install of Koha, which takes finding a spare (virtual) host somewhere. Hrrmmm ... this brings up another topic: Koha dev environments. But I'll leave that for another posting. cheers rickw -- _________________________________ Rick Welykochy || Praxis Services You go find out what they need and I'll start coding. -- two geeks
A quick look over your table on the wiki. One thing I notice is the lack of the marc column. Is this intentional? I don't know if duplicating the marc in two columns is wanted, but I don't like it much. The database, and probably some sql query results, would be much smaller with just one copy on the marc. While we are talking about merging and cleaning in the database can we talk about this also? Thanks. On Sat, 2008-11-22 at 12:58 -0500, Ryan Higgins wrote:
Late entry, I know, but I would like to get rid of the clunky extra table, and merge the columns that we need from biblioitems into a new biblio table. I've listed the columns to keep on the wiki; please review and add any that are missing.
http://wiki.koha.org/doku.php?id=en:development:rfcs3.2:rfc32_biblio_table
-- Ryan Higgins
LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
Yes, the marc column was excluded intentionally, since marcxml is authoritative. I don't really like keeping two copies in there, but if it's useful, we can keep it. As far as I know, Koha uses marcxml exclusively internally now, and the only reason to keep marc would be for faster dumps (though I'd rather dump the authoritative version of the data) What do others think ? Ryan On Mon, Nov 24, 2008 at 10:31 AM, Michael Hafen <mdhafen@tech.washk12.org>wrote:
A quick look over your table on the wiki. One thing I notice is the lack of the marc column. Is this intentional?
I don't know if duplicating the marc in two columns is wanted, but I don't like it much. The database, and probably some sql query results, would be much smaller with just one copy on the marc.
While we are talking about merging and cleaning in the database can we talk about this also?
Thanks.
On Sat, 2008-11-22 at 12:58 -0500, Ryan Higgins wrote:
Late entry, I know, but I would like to get rid of the clunky extra table, and merge the columns that we need from biblioitems into a new biblio table. I've listed the columns to keep on the wiki; please review and add any that are missing.
http://wiki.koha.org/doku.php?id=en:development:rfcs3.2:rfc32_biblio_table
-- Ryan Higgins
LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Ryan Higgins LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704
Given the rarity of dumping MARC data, and the preference for that being the authoritative data, I see no reason to retain the non-XML version. Dumps would take a lot more clock-time to reassemble the MARC machine-code lines, but such operations on more than a handful of records would be (1) rare and (2) on the command-line, where starting off a long job does not cause the appearance of "failure". Dumps of marcxml would be just as fast, of course. --Joe On Mon, Nov 24, 2008 at 11:52 AM, Ryan Higgins <ryan.higgins@liblime.com>wrote:
Yes, the marc column was excluded intentionally, since marcxml is authoritative. I don't really like keeping two copies in there, but if it's useful, we can keep it. As far as I know, Koha uses marcxml exclusively internally now, and the only reason to keep marc would be for faster dumps (though I'd rather dump the authoritative version of the data)
What do others think ?
Ryan
On Mon, Nov 24, 2008 at 10:31 AM, Michael Hafen <mdhafen@tech.washk12.org>wrote:
A quick look over your table on the wiki. One thing I notice is the lack of the marc column. Is this intentional?
I don't know if duplicating the marc in two columns is wanted, but I don't like it much. The database, and probably some sql query results, would be much smaller with just one copy on the marc.
While we are talking about merging and cleaning in the database can we talk about this also?
Thanks.
On Sat, 2008-11-22 at 12:58 -0500, Ryan Higgins wrote:
Late entry, I know, but I would like to get rid of the clunky extra table, and merge the columns that we need from biblioitems into a new biblio table. I've listed the columns to keep on the wiki; please review and add any that are missing.
http://wiki.koha.org/doku.php?id=en:development:rfcs3.2:rfc32_biblio_table
-- Ryan Higgins
LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Ryan Higgins
LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
There is actually one use case where a lot of marc format records are exported frequently. My schools here often export their biblio's and holding's for upload to Follett's TitleWave service for collection analysis. This isn't a big deal, I'm sure I can teach my librarians to wait a little longer for their export. Just so you know. On Mon, 2008-11-24 at 12:23 -0500, Joe Atzberger wrote:
Given the rarity of dumping MARC data, and the preference for that being the authoritative data, I see no reason to retain the non-XML version. Dumps would take a lot more clock-time to reassemble the MARC machine-code lines, but such operations on more than a handful of records would be (1) rare and (2) on the command-line, where starting off a long job does not cause the appearance of "failure". Dumps of marcxml would be just as fast, of course.
--Joe
On Mon, Nov 24, 2008 at 11:52 AM, Ryan Higgins <ryan.higgins@liblime.com> wrote:
Yes, the marc column was excluded intentionally, since marcxml is authoritative. I don't really like keeping two copies in there, but if it's useful, we can keep it. As far as I know, Koha uses marcxml exclusively internally now, and the only reason to keep marc would be for faster dumps (though I'd rather dump the authoritative version of the data)
What do others think ?
Ryan
While we are talking about changes to the biblio table I would like to propose a change. Looking at the file installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql I see that the subfield 245$b is linked to the koha table bibliosubtitle.subtitle, which seems to not exist anymore. I'm sure this is left over from Koha 2.2. I remember this table from before, and so do my librarians. My librarians ask that the marc21 field 245$b be shown on the search results page. I found that by adding a column to the biblio table and linking it in the marc structure I could get the value to export to the template. This happened even though the column in the database remains empty. I can submit a patch to add a column to the biblio table to this end. I'm thinking it would be linked in marc21 to the 245$b, as I have said. In Unimarc it seems 200$e is a good candidate. I don't know Unimarc, and so I hope someone who does can contribute to this RFC a recommendation on which subfield to use there. Once I have community support I will get on the wiki for updatedatabase.pl and reserve an update number for this patch. The name of the column could be either `subtitle` or `remainderoftitle`. The `remainderoftitle` name represents the use of the column in respect to Marc21. The `subtitle` name is more historic, and is probably based on the expectation of linking this field in Unimarc to 200$e (subtitle). Looking at 2.2.9 code though I don't see bibliosubtitle.subtitle linked in the Unimarc structure. I'm leaning toward `subtitle` because it is shorter. However I'm open to suggestions here. The final detail is whether I should put together a script to go in misc which would get values from the marc to fill in the column of the database for existing records. On Sat, 2008-11-22 at 12:58 -0500, Ryan Higgins wrote:
Late entry, I know, but I would like to get rid of the clunky extra table, and merge the columns that we need from biblioitems into a new biblio table. I've listed the columns to keep on the wiki; please review and add any that are missing.
http://wiki.koha.org/doku.php?id=en:development:rfcs3.2:rfc32_biblio_table
-- Ryan Higgins
LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
Michael Hafen a écrit :
While we are talking about changes to the biblio table I would like to propose a change.
Looking at the file installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql I see that the subfield 245$b is linked to the koha table bibliosubtitle.subtitle, which seems to not exist anymore. I'm sure this is left over from Koha 2.2. I remember this table from before, and so do my librarians.
you're right. And you're right it can be removed (although I think it's harmless, and does nothing) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc NOUVEAU TELEPHONE : 04 91 81 35 08
I would like to change this rather than remove it. I would like to add a column to the biblio table to replace bibliosubtitle.subtitle. My librarians like seeing the subtitle in search results and such, but I need to have a database column to reference for this. On Tue, 2008-11-25 at 09:49 +0100, Paul POULAIN wrote:
Michael Hafen a écrit :
While we are talking about changes to the biblio table I would like to propose a change.
Looking at the file installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql I see that the subfield 245$b is linked to the koha table bibliosubtitle.subtitle, which seems to not exist anymore. I'm sure this is left over from Koha 2.2. I remember this table from before, and so do my librarians.
you're right. And you're right it can be removed (although I think it's harmless, and does nothing) -- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
Adding subtitle is ok by me. We (LibLime) are using xslt pretty exclusively, and once we move to the DOM indexing model, we can probably get the for-display record directly from zebra instead of parsing it from the db. But go ahead and add it to the wiki. Ryan On Tue, Nov 25, 2008 at 12:01 PM, Michael Hafen <mdhafen@tech.washk12.org>wrote:
I would like to change this rather than remove it. I would like to add a column to the biblio table to replace bibliosubtitle.subtitle.
My librarians like seeing the subtitle in search results and such, but I need to have a database column to reference for this.
Michael Hafen a écrit :
While we are talking about changes to the biblio table I would like to propose a change.
Looking at the file
installer/data/mysql/en/marcflavour/marc21/mandatory/marc21_framework_DEFAULT.sql I see that the subfield 245$b is linked to the koha table bibliosubtitle.subtitle, which seems to not exist anymore. I'm sure this is left over from Koha 2.2. I remember this table from before, and so do my
On Tue, 2008-11-25 at 09:49 +0100, Paul POULAIN wrote: librarians.
you're right. And you're right it can be removed (although I think it's harmless, and does nothing)
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Ryan Higgins LibLime * Open-Source Solutions for Libraries Featuring KohaZOOM ILS 888-564-2457 x704
I've added this to the list of RFC's. There are still two questions to address I think: Which UNIMARC subfield to link to, and Whether to include a command line script to fill in the database column from the marc data. The unimarc framework I was looking at was incorrect, as there was an update pushed through git which corrected it. I now have no idea which subfield in unimarc this should be linked to. The command line script doesn't seem necessary, but if there is interest I can include one in the patch. If there is no interest I won't since it doesn't seem necessary. It would be good to have the database and the marc the same, but I need a little motivation. ;) Looking forward to more comments. Thank you. On Tue, 2008-11-25 at 13:36 -0500, Ryan Higgins wrote:
Adding subtitle is ok by me. We (LibLime) are using xslt pretty exclusively, and once we move to the DOM indexing model, we can probably get the for-display record directly from zebra instead of parsing it from the db. But go ahead and add it to the wiki.
Ryan
Excellent, and it looks like it isn't used at the moment. Thanks. On Mon, 2008-12-01 at 20:58 +0100, Frederic Demians wrote:
Which UNIMARC subfield to link to, and
In UNIMARC, sub-title field: 200$e.
-- Frédéric -- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
Michael, Have you experimented with the XSL support? I think it's a much better way of solving this problem and many other problems with what should display than adding more columns to the biblio table. My two cents ... Josh On Mon, Dec 1, 2008 at 4:23 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
Excellent, and it looks like it isn't used at the moment.
Thanks.
On Mon, 2008-12-01 at 20:58 +0100, Frederic Demians wrote:
Which UNIMARC subfield to link to, and
In UNIMARC, sub-title field: 200$e.
-- Frédéric -- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Joshua Ferraro SUPPORT FOR OPEN-SOURCE SOFTWARE CEO migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
I looked at it briefly for a different field, and found my understanding of XSL lacking. Adding a column to the database was the quick solution for me. I guess I can take another look. It might be a good solution. My thinking is that it's good to see data from the two ( potentially different ) copies of the record, what's in the database columns and what's in the marcxml column. I'll have to look again at the XSL to see where it comes from too. On Mon, 2008-12-01 at 17:44 -0500, Joshua Ferraro wrote:
Michael,
Have you experimented with the XSL support? I think it's a much better way of solving this problem and many other problems with what should display than adding more columns to the biblio table.
My two cents ...
Josh
On Mon, Dec 1, 2008 at 4:23 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
Excellent, and it looks like it isn't used at the moment.
Thanks.
On Mon, 2008-12-01 at 20:58 +0100, Frederic Demians wrote:
Which UNIMARC subfield to link to, and
In UNIMARC, sub-title field: 200$e.
-- Frédéric -- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
On Mon, Dec 1, 2008 at 6:55 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
I looked at it briefly for a different field, and found my understanding of XSL lacking. Adding a column to the database was the quick solution for me. XSL would also solve the specific problem you have, just turn on those two sysprefs and walla ...
I guess I can take another look. It might be a good solution. My thinking is that it's good to see data from the two ( potentially different ) copies of the record, what's in the database columns and what's in the marcxml column. I'll have to look again at the XSL to see where it comes from too. marcxml has a full MARCXML record blob representing the entire bibliographic and item information for that record. XSL has the advantage of being able to handle repeatable fields, wheras the current database design for Koha can only handle non-repeatable fields.
Josh
On Mon, 2008-12-01 at 17:44 -0500, Joshua Ferraro wrote:
Michael,
Have you experimented with the XSL support? I think it's a much better way of solving this problem and many other problems with what should display than adding more columns to the biblio table.
My two cents ...
Josh
On Mon, Dec 1, 2008 at 4:23 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
Excellent, and it looks like it isn't used at the moment.
Thanks.
On Mon, 2008-12-01 at 20:58 +0100, Frederic Demians wrote:
Which UNIMARC subfield to link to, and
In UNIMARC, sub-title field: 200$e.
-- Frédéric -- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
-- Joshua Ferraro SUPPORT FOR OPEN-SOURCE SOFTWARE CEO migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
On Mon, 2008-12-01 at 19:31 -0500, Joshua Ferraro wrote:
On Mon, Dec 1, 2008 at 6:55 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
I looked at it briefly for a different field, and found my understanding of XSL lacking. Adding a column to the database was the quick solution for me. XSL would also solve the specific problem you have, just turn on those two sysprefs and walla ...
Well, it's not actually that straight forward. I still have to learn how to read and modify an XSLT stylesheet.
I guess I can take another look. It might be a good solution. My thinking is that it's good to see data from the two ( potentially different ) copies of the record, what's in the database columns and what's in the marcxml column. I'll have to look again at the XSL to see where it comes from too. marcxml has a full MARCXML record blob representing the entire bibliographic and item information for that record. XSL has the advantage of being able to handle repeatable fields, wheras the current database design for Koha can only handle non-repeatable fields.
It did occur to me that the database can only handle non-repeatable fields. Luckily for me 245$b is non-repeatable in the Koha Default Biblio Template. Still I think it would be possible to handle repeatable fields, just concatenate the fields together with a seperator, but the database columns would have to be very large to do it right. That isn't really a good solution. At any rate I will take a look at the XSL files. This raises the question of how many others are interested in a patch to this end. Ryan may want to comment here as he has already stated that LibLime uses XSLT. Any others want to comment?
Josh
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
On Tue, Dec 2, 2008 at 10:37 AM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
On Mon, 2008-12-01 at 19:31 -0500, Joshua Ferraro wrote:
On Mon, Dec 1, 2008 at 6:55 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
I looked at it briefly for a different field, and found my understanding of XSL lacking. Adding a column to the database was the quick solution for me. XSL would also solve the specific problem you have, just turn on those two sysprefs and walla ...
Well, it's not actually that straight forward. I still have to learn how to read and modify an XSLT stylesheet. It is if all you want to solve is 245$b ... that's already in the stylesheet.
Cheers, Josh
I guess I can take another look. It might be a good solution. My thinking is that it's good to see data from the two ( potentially different ) copies of the record, what's in the database columns and what's in the marcxml column. I'll have to look again at the XSL to see where it comes from too. marcxml has a full MARCXML record blob representing the entire bibliographic and item information for that record. XSL has the advantage of being able to handle repeatable fields, wheras the current database design for Koha can only handle non-repeatable fields.
It did occur to me that the database can only handle non-repeatable fields. Luckily for me 245$b is non-repeatable in the Koha Default Biblio Template. Still I think it would be possible to handle repeatable fields, just concatenate the fields together with a seperator, but the database columns would have to be very large to do it right. That isn't really a good solution.
At any rate I will take a look at the XSL files.
This raises the question of how many others are interested in a patch to this end. Ryan may want to comment here as he has already stated that LibLime uses XSLT. Any others want to comment?
Josh
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
-- Joshua Ferraro SUPPORT FOR OPEN-SOURCE SOFTWARE CEO migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
I've taken another look at the xslt, specifically for the OPAC search results. The 245$b is already displayed by the stylesheet, but the item call numbers are not. This was the problem I couldn't solve the first time I looked at the xslt. Also it looks to me like the OPAC xslt will only show item availability for the first branch that has an item. Maybe I'm understanding the xslt wrong, but looking at a search result in the OPAC it seems this is true. I'm looking at this bit in the OPAC stylesheet for MARC21: <xsl:variable name="available_items" select="key('item-by-status', 'available')"/> <xsl:for-each select="$available_items[generate-id() = generate-id(key('item-by-status-and-branch', concat(items:status, ' ', items:homebranch))[1])]"> : I don't see how that will grab more that just the first homebranch. I would very much welcome any suggestions on getting item call numbers into the xslt. I'd also welcome comments on the item availability thing. On Tue, 2008-12-02 at 10:40 -0500, Joshua Ferraro wrote:
On Tue, Dec 2, 2008 at 10:37 AM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
On Mon, 2008-12-01 at 19:31 -0500, Joshua Ferraro wrote:
On Mon, Dec 1, 2008 at 6:55 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
I looked at it briefly for a different field, and found my understanding of XSL lacking. Adding a column to the database was the quick solution for me. XSL would also solve the specific problem you have, just turn on those two sysprefs and walla ...
Well, it's not actually that straight forward. I still have to learn how to read and modify an XSLT stylesheet. It is if all you want to solve is 245$b ... that's already in the stylesheet.
Cheers,
Josh
I guess I can take another look. It might be a good solution. My thinking is that it's good to see data from the two ( potentially different ) copies of the record, what's in the database columns and what's in the marcxml column. I'll have to look again at the XSL to see where it comes from too. marcxml has a full MARCXML record blob representing the entire bibliographic and item information for that record. XSL has the advantage of being able to handle repeatable fields, wheras the current database design for Koha can only handle non-repeatable fields.
It did occur to me that the database can only handle non-repeatable fields. Luckily for me 245$b is non-repeatable in the Koha Default Biblio Template. Still I think it would be possible to handle repeatable fields, just concatenate the fields together with a seperator, but the database columns would have to be very large to do it right. That isn't really a good solution.
At any rate I will take a look at the XSL files.
This raises the question of how many others are interested in a patch to this end. Ryan may want to comment here as he has already stated that LibLime uses XSLT. Any others want to comment?
Josh
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://koha-dev.washk12.org or git://koha-dev.washk12.org/koha
Hi, On Tue, Dec 16, 2008 at 5:27 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
Also it looks to me like the OPAC xslt will only show item availability for the first branch that has an item. Maybe I'm understanding the xslt wrong, but looking at a search result in the OPAC it seems this is true.
I'm looking at this bit in the OPAC stylesheet for MARC21: <xsl:variable name="available_items" select="key('item-by-status', 'available')"/> <xsl:for-each select="$available_items[generate-id() = generate-id(key('item-by-status-and-branch', concat(items:status, ' ', items:homebranch))[1])]"> : I don't see how that will grab more that just the first homebranch.
That clause really does scan all of the available items and counts the number of available items for each home branch. It uses the Muenchian method to do the grouping; see http://www.jenitennison.com/xslt/grouping/muenchian.html for an explanation of the general technique. In this case, the XSLT is: [1] Putting a list of all available item nodes into the $available_items variable by doing a lookup on the item-by-status key, which is defined in line 10 of the stylesheet: <xsl:key name="item-by-status" match="items:item" use="items:status"/> [2] Selecting each node in $available_items that is the first available item for each homebranch; this uses the item-by-status-and-branch key. [3] Emitting the homebranch and a count of available items in that homebranch, again using the item-by-status-and-branhc key.
I would very much welcome any suggestions on getting item call numbers into the xslt. I'd also welcome comments on the item availability thing.
You can get the item call numbers in one of two ways: [1] Update the XSLT to grab it from the 952$o, assuming you're using the default MARC21 frameworks. [2] Change C4::XSLT::buildKohaItemsNamespace() to include itemcallnumber in the item XML, then adjust the XSL to use it. Regards, Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt
participants (10)
-
Andrew Moore -
Chris Cormack -
Frederic Demians -
Galen Charlton -
Joe Atzberger -
Joshua Ferraro -
Michael Hafen -
Paul POULAIN -
Rick Welykochy -
Ryan Higgins