On the way to the Debian package
Hi, I think all Koha dependencies are in Debian now (until you add new ones ;-) ). So I started to work on koha packaging itself. My work can be see here: http://git.debian.org/?p=collab-maint/koha.git;a=summary (more exactly here: http://git.debian.org/?p=collab-maint/koha.git;a=shortlog;h=refs/heads/wip ) For now, I work on the 'wip' branch that have frequent rebase. So, do not merge this branch with any other work please. On the way to create the package, I find several things that I would like to see upstream (instead of a diff for the Debian package). This is: - support for DESTDIR at install time - support for the absence of the koha user and group at install time - fix permissions and shebang line when needed - strange file to remove In fact, these are all my changeset but the last (which has debian work in progress, not ready to be commited upstream) So, can you review these changesets and apply them (or something similar) or tell me what is wrong with them ? Another problem of the current code is that all manpages generated from koha perl modules are invalid. They are missing the required head lines. I put in http://www-id.imag.fr/~danjean/lintian.txt the warnings of the lintian tool (automatic debian tool to check package quality). Do you think it is better to fix these doc (ie adding the few lines in each file) or to remove this doc from the Debian package ? I think that fixing is better, but it cannot be done automatically: a 'brief description' is needed so it must be filled by people knowing the modules. Next, I would like to talk about koha-tmpl/*-tmpl/prog/"lg-LG" directories. I see two things: - there are lots of duplicated files and directories. In the Debian packaging, I'm trying to add a 'common' directory and add symlink from the lang directories. This cannot be done in the upstream git repo as some plateform does not support symlinks. But, if files are really similar (for example all libs/ and js/ directories in koha-tmpl/intranet-tmpl/prog/*), they can be moved elsewhere (a common place) and reference in other files could be adjusted ? - koha-tmpl/*-tmpl/prog/"lg-LG"/{lib,js} contains version of other softwares already packaged by debian (such as libjs-jquery or tinymce2 for example). The final debian package will need to use these packages and not duplicate the code by embeding another version. So, I would like to know * if these codes of other applications have been modified (I hope not) * where they are called from (so that I can change to call debian package codes) It all for now. I will come back soon with questions about the configuration :-) Best regards, Vincent PS: I do not fully follow Koha ML. I mostly read messages with 'debian' in the subject, so please try to keep this word in the subject when you want to talk with me. -- Vincent Danjean Adresse: Laboratoire d'Informatique de Grenoble Téléphone: +33 4 76 61 20 11 ENSIMAG - antenne de Montbonnot Fax: +33 4 76 61 20 99 ZIRST 51, avenue Jean Kuntzmann Email: Vincent.Danjean@imag.fr 38330 Montbonnot Saint Martin
Hi, On Fri, Aug 15, 2008 at 4:57 PM, Vincent Danjean <vdanjean.ml@free.fr> wrote:
On the way to create the package, I find several things that I would like to see upstream (instead of a diff for the Debian package). This is: ... So, can you review these changesets and apply them (or something similar) or tell me what is wrong with them ?
I will apply most of these to the Koha 3.1/3.2 HEAD tomorrow. It may take a bit longer for them to reach the 3.0 maintenance branch. Note that the big permissions patch will not apply as is, since the translated templates are not stored in the git tree but are generated during packaging, but I'll add a substitute.
Another problem of the current code is that all manpages generated from koha perl modules are invalid. They are missing the required head lines. I put in http://www-id.imag.fr/~danjean/lintian.txt the warnings of the lintian tool (automatic debian tool to check package quality). Do you think it is better to fix these doc (ie adding the few lines in each file) or to remove this doc from the Debian package ? I think that fixing is better, but it cannot be done automatically: a 'brief description' is needed so it must be filled by people knowing the modules.
Fixing the POD is better.
Next, I would like to talk about koha-tmpl/*-tmpl/prog/"lg-LG" directories. I see two things: - there are lots of duplicated files and directories. In the Debian packaging, I'm trying to add a 'common' directory and add symlink from the lang directories. This cannot be done in the upstream git repo as some plateform does not support symlinks. But, if files are really similar (for example all libs/ and js/ directories in koha-tmpl/intranet-tmpl/prog/*), they can be moved elsewhere (a common place) and reference in other files could be adjusted ?
This is a good idea, but might take a while to be implemented.
- koha-tmpl/*-tmpl/prog/"lg-LG"/{lib,js} contains version of other softwares already packaged by debian (such as libjs-jquery or tinymce2 for example). The final debian package will need to use these packages and not duplicate the code by embeding another version. So, I would like to know * if these codes of other applications have been modified (I hope not)
I'm reasonably sure that we're using stock JQuery, TinyMCE, and YUI. I'm less sure about the JQuery plugins, but I don't see that any of them are currently packaged.
* where they are called from (so that I can change to call debian package codes)
They're generally invoked via the templates, in lines such as <script type="text/javascript" src="<!-- TMPL_VAR name="themelang" -->/lib/greybox/AJS.js"></script> with the exception of YUI: <script type="text/javascript" src="<!-- TMPL_VAR NAME="yuipath" -->/menu/menu-min.js"></script> The themelang and yuipath template variables are ultimately set in the call to C4::Output::gettemplate(). The problem is that a *lot* of the HTML templates invoke JavaScript libraries in that fashion, so if the packaging rules allowing setting up symlinks to the canonical locations of the libraries, that may be simplest. Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt
On Tue, Aug 19, 2008 at 6:26 PM, Galen Charlton <galen.charlton@liblime.com> wrote:
Another problem of the current code is that all manpages generated from koha perl modules are invalid. They are missing the required head lines. I put in http://www-id.imag.fr/~danjean/lintian.txt the warnings of the lintian tool (automatic debian tool to check package quality). Do you think it is better to fix these doc (ie adding the few lines in each file) or to remove this doc from the Debian package ? I think that fixing is better, but it cannot be done automatically: a 'brief description' is needed so it must be filled by people knowing the modules.
Fixing the POD is better.
I agree. I've opened up bug 2538 for this work <http://bugs.koha.org/cgi-bin/bugzilla/show_bug.cgi?id=2538>. extra points for anyone who can write a test case that looks for files missing =NAME tags. -Andy
Hi, Thanks for your replies. Here are some comments and then a few more questions. Galen Charlton wrote:
I will apply most of these to the Koha 3.1/3.2 HEAD tomorrow. It may take a bit longer for them to reach the 3.0 maintenance branch. Note that the big permissions patch will not apply as is, since the translated templates are not stored in the git tree but are generated during packaging, but I'll add a substitute.
Thank you very much. I'm not in a hurry to have them in the 3.0 branch: I can work on my local modified copy. But it is not the kind of patch that I would like to maintain in the Debian package. So, knowing it will be present upstream in a future release is perfect for me.
Another problem of the current code is that all manpages generated from koha perl modules are invalid. They are missing the required head lines. [...]
Fixing the POD is better.
I made a little test on one file (the first one). Applying this kind of change seem to be correct: ===================== diff -u AuthoritiesMarc.pm.orig AuthoritiesMarc.pm --- AuthoritiesMarc.pm.orig 2008-08-20 15:44:45.000000000 +0200 +++ AuthoritiesMarc.pm 2008-08-20 15:50:13.000000000 +0200 @@ -60,6 +60,12 @@ ); } +=head1 NAME + +C4::AuthoritiesMarc - useful module to do something + +=head1 DESCRIPTION + =head2 GetAuthMARCFromKohaField =over 4 ===================== You can easily check if it is good by running: pod2man AuthoritiesMarc.pm > /tmp/C4::AuthoritiesMarc.3pm lexgrog /tmp/C4::AuthoritiesMarc.3pm lexgrog should show you the 'NAME' line (and not 'parse failed' as currently) Andrew: lexgrog can perhaps be used on the generated manpage in order to have a test ?
Next, I would like to talk about koha-tmpl/*-tmpl/prog/"lg-LG" directories. I see two things: - there are lots of duplicated files and directories. In the Debian packaging, I'm trying to add a 'common' directory and add symlink from the lang directories. This cannot be done in the upstream git repo as some plateform does not support symlinks. But, if files are really similar (for example all libs/ and js/ directories in koha-tmpl/intranet-tmpl/prog/*), they can be moved elsewhere (a common place) and reference in other files could be adjusted ?
This is a good idea, but might take a while to be implemented.
Ok. So, for now, in the Debian package, I stick with the 'common' directory and symlinks.
- koha-tmpl/*-tmpl/prog/"lg-LG"/{lib,js} contains version of other softwares already packaged by debian (such as libjs-jquery or tinymce2 for example). The final debian package will need to use these packages and not duplicate the code by embeding another version. So, I would like to know * if these codes of other applications have been modified (I hope not)
I'm reasonably sure that we're using stock JQuery, TinyMCE, and YUI. I'm less sure about the JQuery plugins, but I don't see that any of them are currently packaged.
Thanks. The version number is not exactly the same but I hope it is only bug fix.
* where they are called from (so that I can change to call debian package codes)
They're generally invoked via the templates, in lines such as
<script type="text/javascript" src="<!-- TMPL_VAR name="themelang" -->/lib/greybox/AJS.js"></script>
with the exception of YUI:
<script type="text/javascript" src="<!-- TMPL_VAR NAME="yuipath" -->/menu/menu-min.js"></script>
The themelang and yuipath template variables are ultimately set in the call to C4::Output::gettemplate().
The problem is that a *lot* of the HTML templates invoke JavaScript libraries in that fashion, so if the packaging rules allowing setting up symlinks to the canonical locations of the libraries, that may be simplest.
I think I will stick with either symlinks or Redirect in the apache config so that these files seems to come from the same place. I just see that the git repo does not have the languages directory. Can you tell me how (which program) generate them when you build the tarball ? Do you think that these duplicate files in language directory are here by intend (ie you think that one day JQuery, TinyMCE, YUI, ... will be different, depending on the language) or they are here by 'accident' ? To ask differently, are you interested in a patch that add another copy of these files in a common location and try to remove all call to the 'lang' directory ? Letter, another patch removing them from the language directories can be made. (but here, I need to know how these language directories are generated, hence my previous question) Then, I've a few questions about the configuration: - is it possible to have the intranet and the opac on the same host and port in two different web server directories (for example /opac and /intranet) ? I do not think so but it would be better for the Debian package (easier to propose a config out-of-the-box that works as koha package scripts do not have the right to modify apache config) - I see in several INSTALL file that the global configuration must be adjusted so that the default SAX parser would be XML::LibXML::SAX::Parser This knid of adjustment cannot be done by an official Debian package (modification of a config file from another package). But I do not understand why you do not simply use: $XML::SAX::ParserPackage = "XML::LibXML::SAX::Parser"; Adding this line in sax_parser_print.pl and sax_parser_test.pl just after use XML::SAX; works (without updating my global config file in /etc/perl/XML/SAX/ParserDetails.ini) But I do not know where this is really used in Koha (I cannot find other "use XML::SAX;" in the sources. Perhaps, adding the line $XML::SAX::ParserPackage = "XML::LibXML::SAX::Parser"; in all programs (or in a file included by all koha programs) can avoid this global modification ? - installation instructions recommends to install all files with owner koha:koha. I do not really see the point here. If I understand correctly, it can be better to run the zebra user from another user than root. In this case, which files/dirs need to be written by zebrasrv (/var/log/koha, /var/run/koha/zebradb, var/lib/koha/zebradb and ... ?) Is there other programs (crontab ? ...) that it would be better to run under the koha user/group and which files would need to be writable ? I'm right to think that koha web scripts (scripts run by apache to display the webpages) do not write anything on the filesystem (only in the db ?) [including the installer ?] - I'm right thinking that it is impossible (or very difficult and needing manual intervention) to switch between: * zebra and non-zebra installation (ie what happens if a config with zebra is changed to a config without zebra or vice-et-versa ?) * UNIMARC and MARC21 in zebra config If this is the case, I will try to make sure that the only way to switch from one config to the other with the Debian package would be to purge it (ie remove all, including the database) Thank for your patience and your replies. I hope to be able to present you a experimental package next week. Best regards, Vincent -- Vincent Danjean Adresse: Laboratoire d'Informatique de Grenoble Téléphone: +33 4 76 61 20 11 ENSIMAG - antenne de Montbonnot Fax: +33 4 76 61 20 99 ZIRST 51, avenue Jean Kuntzmann Email: Vincent.Danjean@imag.fr 38330 Montbonnot Saint Martin
Vincent -- Thanks for your work on this. I think I can answer a couple of your questions. Do you think that these duplicate files in language directory are here
by intend (ie you think that one day JQuery, TinyMCE, YUI, ... will be different, depending on the language) or they are here by 'accident' ? To ask differently, are you interested in a patch that add another copy of these files in a common location and try to remove all call to the 'lang' directory ?
The external js libraries, to my knowlege, are not translated and probably should not be translated. A "common" directory would make a lot of sense for them (and possibly for some icon sets too). There's been discussion of doing this since 2007 at least.
Then, I've a few questions about the configuration: - is it possible to have the intranet and the opac on the same host and port in two different web server directories (for example /opac and /intranet) ?
As it stands, there would be conflict with this type of installation. In particular, the session cookies would overwrite each other if a user logged into both at the same time. --Joe Atzberger, LibLime
Vincent --
Thanks for your work on this. I think I can answer a couple of your questions.
Do you think that these duplicate files in language directory are here by intend (ie you think that one day JQuery, TinyMCE, YUI, ... will be different, depending on the language) or they are here by 'accident' ? To ask differently, are you interested in a patch that add another copy of these files in a common location and try to remove all call to the 'lang' directory ?
The external js libraries, to my knowlege, are not translated and probably should not be translated. A "common" directory would make a lot of sense for them (and possibly for some icon sets too). There's been discussion of doing this since 2007 at least.
Then, I've a few questions about the configuration: - is it possible to have the intranet and the opac on the same host and port in two different web server directories (for example /opac and /intranet) ?
As it stands, there would be conflict with this type of installation. In particular, the session cookies would overwrite each other if a user logged into both at the same time. From what I know, "cart" was directly used from yui and there was a
Joe Atzberger a écrit : translation issue for that. And we had to translate the js lib but for one word, I had rather set the name in the templates.
From what I know, "cart" was directly used from yui and there was a translation issue for that. And we had to translate the js lib but for one word, I had rather set the name in the templates.
Henri -- I think you are talking about translating Koha's js that in turn uses YUI, not translating the YUI js library itself. If YUI itself required translation, then none of the libraries using translations would be able to use the externally hosted YUI libs. It would also be something of a defect in Yahoo's design. --joe
Hi, Always preparing the Debian package, I see there are lots of things installed in usr/share/koha/{bin,misc}. I would like to know what are these files. In particular, are they - files/scripts invoked by koha[1] - files/scripts needed to run zebra - files/scripts that can help a user in special occasion (will be probably moved in /usr/share/doc/koha/examples) - files/scripts intend to be run by the admin (will be probably moved in /usr/sbin) - ... [1]: when I say "run by koha", I mean "run by the cgi-scripts by apache" So, here is the (long) list with some of my comments. Please, correct me or add any information you know. * bin directory with all installed executables of koha (files will probably be dispached in the debian packages) * bin/batchCompareMARCvsFrameworks.pl * bin/batchDeleteUnusedSubfields.pl * bin/batchImportMARCWithBiblionumbers.pl * bin/batchRebuildBiblioTables.pl * bin/batchRepairMissingBiblionumbers.pl * bin/batchupdateISBNs.pl * bin/check_sysprefs.pl * bin/commit_biblios_file.pl * bin/exportauth.pl * bin/link_bibs_to_authorities.pl * bin/spellcheck_suggest * bin/spellcheck_suggest/make_spellcheck_suggest.pl * bin/stage_biblios_file.pl * bin/syspref-diff.pl * bin/zebraqueue_daemon.pl ?? run by koha ? * bin/perlmodule_ls.pl * bin/perlmodule_rm.pl * bin/sax_parser_print.pl * bin/sax_parser_test.pl * bin/testKoha.pl Really useful after an installation ? * bin/kohalib.pl helper script for the others. => others will need patching (for PERL5LIB) if they are moved in the debian package. * bin/koha-pazpar2-ctl.sh * bin/koha-zebra-ctl.sh * bin/koha-zebraqueue-ctl.sh /etc/init.d/ scripts. Why two different for zebra ? Can they be merged ? * bin/cronjobs cronjobs to do some things. But which ? Should they be run by default ? Which frequency ? Under which user ? (ie which special file do they need to read (such as /etc/koha/koha-conf.xml) ?) * bin/cronjobs/advance_notices.pl * bin/cronjobs/backup.sh * bin/cronjobs/build_browser_and_cloud.pl * bin/cronjobs/crontab.example has some minimal doc wrt to some of the other cron files => can benefice to be rewritten by rewrite.PL at install time (for some paths) * bin/cronjobs/fines.pl * bin/cronjobs/j2a.pl * bin/cronjobs/longoverdue.pl * bin/cronjobs/notifyMailsOp.pl * bin/cronjobs/overdue_notices.pl * bin/cronjobs/process_message_queue.pl * bin/cronjobs/reservefix.pl * bin/cronjobs/services_throttle.pl * bin/cronjobs/smsoverdues.pl * bin/cronjobs/stats * bin/cronjobs/stats/monthly_circulation_statistics.pl * bin/cronjobs/stats/monthly_new_items_statistics.pl * bin/cronjobs/stats/monthly_new_patron_statistics.pl * bin/cronjobs/update_items.pl * bin/cronjobs/zebraqueue_start.pl * bin/load_testing * bin/load_testing/benchmark_circulation.pl * bin/load_testing/benchmark_webservices.pl ??? * bin/maintenance * bin/maintenance/fix_accountlines_date.pl * bin/maintenance/sync_items_in_marc_bib.pl ??? * bin/migration_tools * bin/migration_tools/22_to_30 * bin/migration_tools/22_to_30/biblio_framework.sql * bin/migration_tools/22_to_30/convert_to_utf8.pl * bin/migration_tools/22_to_30/export_Authorities.pl * bin/migration_tools/22_to_30/export_Authorities_xml.pl * bin/migration_tools/22_to_30/missing090field.pl * bin/migration_tools/22_to_30/move_marc_to_authheader.pl * bin/migration_tools/22_to_30/move_marc_to_biblioitems.pl * bin/migration_tools/22_to_30/phrase_log.sql * bin/migration_tools/22_to_30/rebuild_leader.pl * bin/migration_tools/22_to_30/rebuild_unimarc_100.pl * bin/migration_tools/build6xx.pl * bin/migration_tools/buildCOUNTRY.pl * bin/migration_tools/buildEDITORS.pl * bin/migration_tools/buildLANG.pl * bin/migration_tools/bulkauthimport.pl * bin/migration_tools/bulkmarcimport.pl * bin/migration_tools/fix_onloan.pl * bin/migration_tools/merge_authority.pl * bin/migration_tools/rebuild_nozebra.pl * bin/migration_tools/rebuild_zebra.pl * bin/migration_tools/upgradeitems.pl Is it needed in a Debian package (ie where no pre 3.0 version exists ?) Perhaps as examples only. Or perhaps 'rebuild_nozebra.pl' and 'rebuild_zebra.pl' can be used to switch between a zebra and nozebra installation ? * misc * misc/koha-install-log Does the installer use this file ? As upgrade will be managed by dpkg, I do not see the interest of this file (moreover, some values will be undefined as the user will have some choices at package install time (unimarc/marc21, ...) and this file is created at package build time) * misc/rss * misc/rss/lastAcquired-1.0.conf * misc/rss/lastAcquired-1.0.tmpl * misc/rss/lastAcquired-2.0.conf * misc/rss/lastAcquired-2.0.tmpl * misc/rss/lastAcquired.conf * misc/rss/lastAcquired.tmpl * misc/rss/longestUnseen.conf * misc/rss/longestUnseen.tmpl * misc/rss/mostReserved.conf * misc/rss/mostReserved.tmpl * misc/rss/README * misc/rss/rss.pl * misc/rss/sm-koha-icon.jpg Are they examples ? Or are they used by koha ? If they are examples, what is needed to use them ? * misc/translator * misc/translator/install-code.pl * misc/translator/install.pl * misc/translator/rebuild_lang.sh * misc/translator/stats.pl * misc/translator/text-extract2.pl * misc/translator/text-extract.pl * misc/translator/tmpl_process3.pl * misc/translator/TmplTokenizer.pm * misc/translator/TmplToken.pm * misc/translator/TmplTokenType.pm * misc/translator/translator_doc.html * misc/translator/translator_doc.txt * misc/translator/update.pl * misc/translator/VerboseWarnings.pm * misc/translator/xgettext.pl Are they really needed on a running system ? (ie if used and the translations changed, all changes will be overwritten by the next package update => better use these tools before or while building the package). Best regards, Vincent
Galen Charlton wrote:
Hi,
On Fri, Aug 15, 2008 at 4:57 PM, Vincent Danjean <vdanjean.ml@free.fr> wrote:
On the way to create the package, I find several things that I would like to see upstream (instead of a diff for the Debian package). This is: ... So, can you review these changesets and apply them (or something similar) or tell me what is wrong with them ?
I will apply most of these to the Koha 3.1/3.2 HEAD tomorrow.
I looked at the translation system. I've a few suggestion of patch for upstream. I put all the changeset that can interest upstream in my debian git tree that has the following branches: upstream: your release wip-for-upstream: upstream + changesets that can be applied upstream (IMHO) wip: wip-for-upstream + changesets leading to a debian package About translation, I've: * Use relative path to avoid people dependent paths Some scripts were using private absolute path. I do not think this is needed * Force ordering of files when creating/updating po files Files were listed in raw (inode) order. So the list was not the same between all machines. This generated different .po files even if nothing changed :-( => now, only POT-Changed-Date is modified when running ./update.pl several times. Note that this is a gettext bug that will be fixed in the next release (see http://bugs.debian.org/496282 and https://savannah.gnu.org/bugs/?24123 for more information) * Remove temporary files when updating po files update.pl was creating lots of files in /tmp. Their removal were commented out. When running ./update.pl and then ./install.pl, I see that the release was not up-to-date wrt translation. Was it intentional or was it a bug ? I plan to rebuild the translated file (by invoking ./update.pl and ./install.pl) in the debian package in order to be able to quickly fix translation errors reported by user (and not to have to wait for a new upstream release of koha with these translation bugs fixed). This is why I would like to know if running ./update.pl and then ./install.pl is the correct thing to do. Or, perhaps, some languages must be manually excluded ? I also run in the po directory the following commands: for f in *.po ; do echo -n "$f: "; msgfmt --statistics -o /dev/null $f ; done => lots of po files are not really up-to-date for f in *.po ; do echo; echo "=> $f: "; msgfmt --statistics -o /dev/null -cv $f ; done => and there are errors in a lot of them What do you think: is it better that the debian package offers as many translations as possible (even if they are incomplete). Or should I create l10n packages only for language with enought translation ? (which limit then ?) Best regards, Vincent
Hi, On Sun, Aug 24, 2008 at 8:52 AM, Vincent Danjean <vdanjean.ml@free.fr> wrote:
What do you think: is it better that the debian package offers as many translations as possible (even if they are incomplete). Or should I create l10n packages only for language with enought translation ? (which limit then ?)
I'd prefer packaging as many translations as possible, on the theory that if there's enough interest for a particular language, an incomplete translation will eventually be improved. Regards, Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt
participants (5)
-
Andrew Moore -
Galen Charlton -
Henri-Damien LAURENT -
Joe Atzberger -
Vincent Danjean