I am part of a team of researchers in the Brigham Young University Computer Science department who are interested in improving the searching used by Koha's OPAC interface. Others on the team have come up with a fast and effective way to do searching using folksonomies and we are interested in adding that system to one of the open-source library systems out there. We are considering both Koha and Evergreen for this. We would like to have the Koha communities involvement as much as possible. We would like to either enhance the existing search algorithm of Koha or replace it with our own search. We would like to know if anyone is interested in being involved with this? We would also like to get any pointers on how possible and how difficult this might be for our team to do this, keeping in mind that we are newish to the library specific protocols and formats (MARC, Z39.50, etc). Thanks for your consideration. -- Thank you, Chad Hansen e. cgh@byu.edu p. 801-477-0295
Koha uses two methods of searching in the OPAC, and in general. The first is through third party open source package from IndexData called Zebra. The second is NoZebra mode. My opinion is that applying your algorithm to the NoZebra mode would be the easiest option here. Unless you want to also work with IndexData to add to that package. I'm not totally familiar with how NoZebra searches work currently, but my understanding is that there is a database table of search terms and the record id's associated with those terms. Others from the list may have a better understanding, or be able to provide more details. Point is that there may need to be some changes to the database as part of implementing your algorithm. This isn't a big deal though, Koha has an upgrade mechanism to handle this. The big catch is that library formats (MARC) will play a big role in what is index for searching. Some limited knowledge will be necessary. This may be provided by members of the Koha developer community though. Just be warned that the various MARC formats are complex. To summarize: adding your algorithm to Koha is possible, and will be easy with help from the developers community. Good luck with your project. On Thu, 2009-12-10 at 10:17 -0700, Chad G. Hansen wrote:
I am part of a team of researchers in the Brigham Young University Computer Science department who are interested in improving the searching used by Koha's OPAC interface. Others on the team have come up with a fast and effective way to do searching using folksonomies and we are interested in adding that system to one of the open-source library systems out there. We are considering both Koha and Evergreen for this. We would like to have the Koha communities involvement as much as possible.
We would like to either enhance the existing search algorithm of Koha or replace it with our own search.
We would like to know if anyone is interested in being involved with this? We would also like to get any pointers on how possible and how difficult this might be for our team to do this, keeping in mind that we are newish to the library specific protocols and formats (MARC, Z39.50, etc).
Thanks for your consideration.
-- Thank you, Chad Hansen
e. cgh@byu.edu p. 801-477-0295 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
NoZebra mode is at a severe disadvantage in terms of featureset, performance, scalability, extensibility and basically everything (except ease of installation). I would not recommend that anyone who wants their feature to be used by even modestly sized libraries should develop it for NoZebra mode. The only widely useful development would be with zebra. To clarify: what they call folksonomy, most people nowadays just call tagging. Koha has tagging already, and just needs to work it into the zebra indexes better. Another valid expansion of this feature would be to integrate with LibraryThing's shared tags metadata. That takes the tags outside the ILS instance and into a separate repository. --joe On Thu, Dec 10, 2009 at 12:28 PM, Michael Hafen <mdhafen@tech.washk12.org>wrote:
Koha uses two methods of searching in the OPAC, and in general. The first is through third party open source package from IndexData called Zebra. The second is NoZebra mode. My opinion is that applying your algorithm to the NoZebra mode would be the easiest option here. Unless you want to also work with IndexData to add to that package.
I'm not totally familiar with how NoZebra searches work currently, but my understanding is that there is a database table of search terms and the record id's associated with those terms. Others from the list may have a better understanding, or be able to provide more details. Point is that there may need to be some changes to the database as part of implementing your algorithm. This isn't a big deal though, Koha has an upgrade mechanism to handle this.
The big catch is that library formats (MARC) will play a big role in what is index for searching. Some limited knowledge will be necessary. This may be provided by members of the Koha developer community though. Just be warned that the various MARC formats are complex.
To summarize: adding your algorithm to Koha is possible, and will be easy with help from the developers community.
Good luck with your project.
On Thu, 2009-12-10 at 10:17 -0700, Chad G. Hansen wrote:
I am part of a team of researchers in the Brigham Young University Computer Science department who are interested in improving the searching used by Koha's OPAC interface. Others on the team have come up with a fast and effective way to do searching using folksonomies and we are interested in adding that system to one of the open-source library systems out there. We are considering both Koha and Evergreen for this. We would like to have the Koha communities involvement as much as possible.
We would like to either enhance the existing search algorithm of Koha or replace it with our own search.
We would like to know if anyone is interested in being involved with this? We would also like to get any pointers on how possible and how difficult this might be for our team to do this, keeping in mind that we are newish to the library specific protocols and formats (MARC, Z39.50, etc).
Thanks for your consideration.
-- Thank you, Chad Hansen
e. cgh@byu.edu p. 801-477-0295 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Thank you for your responses. The design of the enhanced library search engine we would like to integrate with Koha can be found in the 2009 JASIST paper which was co-authored by several members of our team now. Here is the reference for that paper: Maria Soledad Pera, William Lund, and Yiu-Kai Ng, A Sophisticated Library Search Strategy Using Kolksonomies and Similarity Matches < http://faculty.cs.byu.edu/~dennis/papers/EnLibS-Rev.pdf<http://faculty.cs.byu.edu/%7Edennis/papers/EnLibS-Rev.pdf>>. Journal of the American Society of Information Science and Technology (JASIST) Volumne 60, Issue 7, pp. 1392-1406, July 2009. You can download the paper from the link and view it if you would like. Essentialy the sustem uses folksonomies (tagging) to provide more relevent search results to the user. The tags they used to do similarity matching were actually from LibraryThing. I would suggest reading at least the abstract of this document if you want to know more of what thy did. I will look into Zebra a little more to see if our answer is with them. Thanks, Chad On Thu, Dec 10, 2009 at 12:03 PM, Joe Atzberger <ohiocore@gmail.com> wrote:
NoZebra mode is at a severe disadvantage in terms of featureset, performance, scalability, extensibility and basically everything (except ease of installation). I would not recommend that anyone who wants their feature to be used by even modestly sized libraries should develop it for NoZebra mode. The only widely useful development would be with zebra.
To clarify: what they call folksonomy, most people nowadays just call tagging. Koha has tagging already, and just needs to work it into the zebra indexes better. Another valid expansion of this feature would be to integrate with LibraryThing's shared tags metadata. That takes the tags outside the ILS instance and into a separate repository.
--joe
On Thu, Dec 10, 2009 at 12:28 PM, Michael Hafen <mdhafen@tech.washk12.org>wrote:
Koha uses two methods of searching in the OPAC, and in general. The first is through third party open source package from IndexData called Zebra. The second is NoZebra mode. My opinion is that applying your algorithm to the NoZebra mode would be the easiest option here. Unless you want to also work with IndexData to add to that package.
I'm not totally familiar with how NoZebra searches work currently, but my understanding is that there is a database table of search terms and the record id's associated with those terms. Others from the list may have a better understanding, or be able to provide more details. Point is that there may need to be some changes to the database as part of implementing your algorithm. This isn't a big deal though, Koha has an upgrade mechanism to handle this.
The big catch is that library formats (MARC) will play a big role in what is index for searching. Some limited knowledge will be necessary. This may be provided by members of the Koha developer community though. Just be warned that the various MARC formats are complex.
To summarize: adding your algorithm to Koha is possible, and will be easy with help from the developers community.
Good luck with your project.
On Thu, 2009-12-10 at 10:17 -0700, Chad G. Hansen wrote:
I am part of a team of researchers in the Brigham Young University Computer Science department who are interested in improving the searching used by Koha's OPAC interface. Others on the team have come up with a fast and effective way to do searching using folksonomies and we are interested in adding that system to one of the open-source library systems out there. We are considering both Koha and Evergreen for this. We would like to have the Koha communities involvement as much as possible.
We would like to either enhance the existing search algorithm of Koha or replace it with our own search.
We would like to know if anyone is interested in being involved with this? We would also like to get any pointers on how possible and how difficult this might be for our team to do this, keeping in mind that we are newish to the library specific protocols and formats (MARC, Z39.50, etc).
Thanks for your consideration.
-- Thank you, Chad Hansen
e. cgh@byu.edu p. 801-477-0295 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
-- Thank you, Chad e. chansen4@gmail.com e. cgh@byu.edu p. 801-477-0295
Maria Soledad Pera, William Lund, and Yiu-Kai Ng, A Sophisticated Library Search Strategy Using Kolksonomies and Similarity Matches <http://faculty.cs.byu.edu/~dennis/papers/EnLibS-Rev.pdf>. Journal of the
"In solving this problem, we propose an enhanced library system, which allows partial, similarity matching of (i) tags defined by ordinary users at a folksonomy site that describe the content of books and (ii) unrestricted keywords specified by an ordinary library patron in a query to search for relevant library catalog records. The proposed library system allows patrons posting a query Q using commonly-used words and ranks the retrieved results according to their degrees of resemblance with Q while maintaining the query processing time comparable with the one achieved by current library search engines." I haven't read the whole thing, but I'm confused by this. Is there some aspect of Koha's tagging function which does not fulfill "( i )" above? Is a full-text search of MARC records (including subject headings, notes, descriptions, etc) not the same as a search of "unrestricted" keywords? This proposal seems to imply a conception of Koha's search function as being limited to subject headings. -- Owen -- Web Developer Athens County Public Libraries http://www.myacpl.org
No, there is no problem with Koha's tagging function. This system is meant to use those tags as an aid in finding more relevant results. Our system uses tags (either from Koha or a third party system) to find more relevant results to return to the user, not just using stuff stored in a MARC record, but tags that ordinary people have used to describe the content of books as well. Here is what we are looking for. My group is submitting a proposal for a grant that will hopefully allow us to implement our new search technology with one of the existing open source ILS that are being used around the world. We would like to work with Koha (and maybe it would be more appropriate to talk to the Zebra developers about this. What do you think?) to see if we can improve its OPAC search with ideas or features from our own system. We are just in the proposal phase right now. Attached is a portion of a previously rejected proposal. The section attached is our previous design/plan section for what we would like to accomplish. What we are looking for is an endorement by the Koha (or Zebra) development community of the idea. A willingness for someone in the development community to possibly work with us in our attempt to potentially improve Koha's search. Our goal is to work with the development community to implement any improvements we can and then give what we have done to the community (we have no desire to maintain after the 3 year grant project is up). Is this the right venue for this request? What is the concensus from this group? Do you need more information before commiting to endoring the idea to possibly work with us in the future (assuming our grant is granted)? Thanks, Chad Hansen On Fri, Dec 18, 2009 at 11:21 AM, Owen Leonard <oleonard@myacpl.org> wrote:
Maria Soledad Pera, William Lund, and Yiu-Kai Ng, A Sophisticated Library Search Strategy Using Kolksonomies and Similarity Matches <http://faculty.cs.byu.edu/~dennis/papers/EnLibS-Rev.pdf<http://faculty.cs.byu.edu/%7Edennis/papers/EnLibS-Rev.pdf>>. Journal of the
"In solving this problem, we propose an enhanced library system, which allows partial, similarity matching of (i) tags defined by ordinary users at a folksonomy site that describe the content of books and (ii) unrestricted keywords specified by an ordinary library patron in a query to search for relevant library catalog records. The proposed library system allows patrons posting a query Q using commonly-used words and ranks the retrieved results according to their degrees of resemblance with Q while maintaining the query processing time comparable with the one achieved by current library search engines."
I haven't read the whole thing, but I'm confused by this. Is there some aspect of Koha's tagging function which does not fulfill "( i )" above?
Is a full-text search of MARC records (including subject headings, notes, descriptions, etc) not the same as a search of "unrestricted" keywords? This proposal seems to imply a conception of Koha's search function as being limited to subject headings.
-- Owen
-- Web Developer Athens County Public Libraries http://www.myacpl.org
-- Thank you, Chad e. chansen4@gmail.com e. cgh@byu.edu p. 801-477-0295
My group is submitting a proposal for a grant that will hopefully allow us to implement our new search technology with one of the existing open source ILS that are being used around the world. We would like to work with Koha (and maybe it would be more appropriate to talk to the Zebra developers about this. What do you think?) to see if we can improve its OPAC search with ideas or features from our own system. We are just in the proposal phase right now. Attached is a portion of a previously rejected proposal. The section attached is our previous design/plan section for what we would like to accomplish. What we are looking for is an endorement by the Koha (or Zebra) development community of the idea. A willingness for someone in the development community to possibly work with us in our attempt to potentially improve Koha's search. Our goal is to work with the development community to implement any improvements we can and then give what we have done to the community (we have no desire to maintain after the 3 year grant project is up). Is this the right venue for this request? What is the concensus from this group? Do you need more information before commiting to endoring the idea to possibly work with us in the future (assuming our grant is granted)? Thanks, Chad Hansen
"Chad G. Hansen" <cgh@byu.edu> wrote:
[...] Attached is a portion of a previously rejected proposal. The section attached is our previous design/plan section for what we would like to accomplish.
Please don't send attachments to mailing lists unless it's usual practice (like sending a patch to a software development list) or you really have no alternative. You will be taking up a lot of space, roughly the size of the attachment multiplied by the number of people on the list. One 3Mb waste is bad. 300 times 3Mb gets painful if common. If it's mainly text, put the text as text in the body of a text email. It's really much easier for more people to read.
What we are looking for is an endorement by the Koha (or Zebra) development community of the idea. A willingness for someone in the development community to possibly work with us in our attempt to potentially improve Koha's search. Our goal is to work with the development community to implement any improvements we can and then give what we have done to the community (we have no desire to maintain after the 3 year grant project is up).
Is this the right venue for this request?
Probably.
What is the concensus from this group?
I'd expect everyone to like exploring search improvements. I'm surprised that this didn't get a reply from someone else while I was snowbound.
Do you need more information before commiting to endoring the idea to possibly work with us in the future (assuming our grant is granted)?
I'd need to know how any workers we'd commit would get paid. Hope that helps, -- MJ Ray (slef) Webmaster and LMS developer at | software www.software.coop http://mjr.towers.org.uk | .... co IMO only: see http://mjr.towers.org.uk/email.html | .... op
Chad, Neat! Always glad to have people volunteer to help make Koha better. When you say "improve" and "enhance" the search, what are your specific goals? Faster retrieval time? More complex search options (nested Booleans, more search fields, regex-style wildcards, etc.)? More relevant relevance ranking? All of the above? Knowing more about the intended result will help us determine how difficult the process might be. Thanks for your willingness to contribute; look forward to hearing more from you! Ian Walls Systems Integration Librarian NYU Health Sciences Libraries 550 First Ave., New York, NY 10016 (212) 263-8687 From: koha-devel-bounces@lists.koha.org [mailto:koha-devel-bounces@lists.koha.org] On Behalf Of Chad G. Hansen Sent: Thursday, December 10, 2009 12:18 PM To: koha-devel@lists.koha.org Subject: [Koha-devel] Improving Koha's OPAC Search I am part of a team of researchers in the Brigham Young University Computer Science department who are interested in improving the searching used by Koha's OPAC interface. Others on the team have come up with a fast and effective way to do searching using folksonomies and we are interested in adding that system to one of the open-source library systems out there. We are considering both Koha and Evergreen for this. We would like to have the Koha communities involvement as much as possible. We would like to either enhance the existing search algorithm of Koha or replace it with our own search. We would like to know if anyone is interested in being involved with this? We would also like to get any pointers on how possible and how difficult this might be for our team to do this, keeping in mind that we are newish to the library specific protocols and formats (MARC, Z39.50, etc). Thanks for your consideration. -- Thank you, Chad Hansen e. cgh@byu.edu<mailto:cgh@byu.edu> p. 801-477-0295 </PRE> <html> <body> ------------------------------------------------------------<br /> This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.<br /> ================================= </body> </html> <PRE>
Ian, The main goal behind the folksonomic aided searching, from what I understand (I was not on the team that developed this searching algorithm, I have been added as someone who can potentially help them get their algorithm into an existing ILS), is to improve relevance of the results. I will talk to my team about this further to determine what improvements we are actually hoping for. Thanks, Chad On Thu, Dec 10, 2009 at 10:35 AM, Walls, Ian <Ian.Walls@med.nyu.edu> wrote:
Chad,
Neat! Always glad to have people volunteer to help make Koha better.
When you say “improve” and “enhance” the search, what are your specific goals? Faster retrieval time? More complex search options (nested Booleans, more search fields, regex-style wildcards, etc.)? More relevant relevance ranking? All of the above? Knowing more about the intended result will help us determine how difficult the process might be.
Thanks for your willingness to contribute; look forward to hearing more from you!
Ian Walls
Systems Integration Librarian
NYU Health Sciences Libraries
550 First Ave., New York, NY 10016
(212) 263-8687
*From:* koha-devel-bounces@lists.koha.org [mailto: koha-devel-bounces@lists.koha.org] *On Behalf Of *Chad G. Hansen *Sent:* Thursday, December 10, 2009 12:18 PM *To:* koha-devel@lists.koha.org *Subject:* [Koha-devel] Improving Koha's OPAC Search
I am part of a team of researchers in the Brigham Young University Computer Science department who are interested in improving the searching used by Koha's OPAC interface. Others on the team have come up with a fast and effective way to do searching using folksonomies and we are interested in adding that system to one of the open-source library systems out there. We are considering both Koha and Evergreen for this. We would like to have the Koha communities involvement as much as possible.
We would like to either enhance the existing search algorithm of Koha or replace it with our own search.
We would like to know if anyone is interested in being involved with this? We would also like to get any pointers on how possible and how difficult this might be for our team to do this, keeping in mind that we are newish to the library specific protocols and formats (MARC, Z39.50, etc).
Thanks for your consideration.
-- Thank you, Chad Hansen
e. cgh@byu.edu p. 801-477-0295
------------------------------------------------------------ This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email. =================================
-- Thank you, Chad e. chansen4@gmail.com e. cgh@byu.edu p. 801-477-0295
participants (6)
-
Chad G. Hansen -
Joe Atzberger -
Michael Hafen -
MJ Ray -
Owen Leonard -
Walls, Ian