Hi I propose to get etc and translator as external git repositories. Why ? It would allow to have po files in their own repository and limit the weight of Koha git archives. For etc, it would allow to change the installer in order to keep track of local changes for every person and therefore be able to synchronize with community version properly. HOW ? operation is simple : git filter-branch --subdirectory-filter misc/translator -- --all and then git push on an external repository. for instance etc.git and translator.git on git.koha-community This would allow to have more than one branch for etc and for instance create and etc/apache etc/nginx etc/plack and so on... WHEN ? when you think operation should be done, if you think it could be interesting. and then we could have git submodules in the koha.git repository in order to keep in synch and synchronise versions when we want. http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html It is just a proposition. -- Henri-Damien LAURENT
I propose to get etc and translator as external git repositories. Why ?
It would allow to have po files in their own repository and limit the weight of Koha git archives. For etc, it would allow to change the installer in order to keep track of local changes for every person and therefore be able to synchronize with community version properly.
Good idea. I don't know if the size of Koha git repository is an issue but for organizational purpose, dividing Koha into smaller parts seems great. Another issue is the fact that we have two Koha downloadable versions, one with translated templates and one without. With translated templates, Koha archive is 241 Mo, when english-only version is 30 Mo. I plan for 3.4, when Template Toolkit will be integrated, to modify the web installer in order to add the ability to pick up a language and to translate it on the fly. This way we would have a unique Koha archive. But we can also have multiple translation packages containing .po files (and Debian packages) installable only when needed. Imagine: apt-get install koha koha-locale-fr A tar gzip of current po directory produces a 21 Mo archive. So I suppose that a Koha archive without any translation at all (no .po files) would weight only 9 Mo. -- Frédéric
Frédéric Demians schreef op wo 10-11-2010 om 17:55 [+0100]:
But we can also have multiple translation packages containing .po files (and Debian packages) installable only when needed. Imagine:
apt-get install koha koha-locale-fr
fwiw, this is exactly my eventual plan for dealing with translations in the packages. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5957 6D23 8B16 EFAB FEF8 7175 14D3 6485 A99C EB6D
Hi, On Wed, Nov 10, 2010 at 11:11 AM, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Hi I propose to get etc and translator as external git repositories. Why ?
It would allow to have po files in their own repository and limit the weight of Koha git archives. For etc, it would allow to change the installer in order to keep track of local changes for every person and therefore be able to synchronize with community version properly.
-1 for moving etc to a separate Git project. The default configuration files are an integral part of Koha. For variant configurations such as a Plack or nginx config, it would be clearer to have relevant files in subdirectories or identified by file name instead of putting them in branches. That way, somebody installing Koha for the first time would see (say) both an etc/koha-apache.conf and an etc/koha-nginx.conf; i.e., all of the options would be directly in front of them. Furthermore, often changes made to files in etc are dependent on changes made to code or templates; separating etc and the rest of Koha into submodules would make it harder to prepare patches and manage topic branches. 0 for moving the PO files to a separate Git project. The size of the repository doesn't really strike me as a big deal; the Git protocol is pretty efficient. That said, while I don't see a great deal of benefit to splitting the translations off into a separate repository, I don't see much harm either. One thing to keep in mind is that whether the release tarballs include all languages expanded or not has nothing to do with how the Git repositories are structured. Looking at the download statistics for koha-3.02.00.tar.gz versus koha-3.02.00-all-translations.tar.gz, very few people use or need the all-translations version. Regards, Galen -- Galen Charlton gmcharlt@gmail.com
On Wed, Nov 10, 2010 at 1:10 PM, Galen Charlton <gmcharlt@gmail.com> wrote:
-1 for moving etc to a separate Git project. The default configuration files are an integral part of Koha. For variant configurations such as a Plack or nginx config, it would be clearer to have relevant files in subdirectories or identified by file name instead of putting them in branches. That way, somebody installing Koha for the first time would see (say) both an etc/koha-apache.conf and an etc/koha-nginx.conf; i.e., all of the options would be directly in front of them. Furthermore, often changes made to files in etc are dependent on changes made to code or templates; separating etc and the rest of Koha into submodules would make it harder to prepare patches and manage topic branches.
I'm with Galen here.
0 for moving the PO files to a separate Git project. The size of the repository doesn't really strike me as a big deal; the Git protocol is pretty efficient. That said, while I don't see a great deal of benefit to splitting the translations off into a separate repository, I don't see much harm either.
If setting up the PO files as a separate project allows one PO repo to be "submoduled" to several Koha repos then I can see how this might benefit a multi-koha server. As it is, we don't implement any language other than English, so it really does not matter to me one way or the other. Kind Regards, Chris
Hi,
On Wed, Nov 10, 2010 at 11:11 AM, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Hi I propose to get etc and translator as external git repositories. Why ?
It would allow to have po files in their own repository and limit the weight of Koha git archives. For etc, it would allow to change the installer in order to keep track of local changes for every person and therefore be able to synchronize with community version properly.
-1 for moving etc to a separate Git project. The default configuration files are an integral part of Koha. For variant configurations such as a Plack or nginx config, it would be clearer to have relevant files in subdirectories or identified by file name instead of putting them in branches. That way, somebody installing Koha for the first time would see (say) both an etc/koha-apache.conf and an etc/koha-nginx.conf; i.e., all of the options would be directly in front of them. Furthermore, often changes made to files in etc are dependent on changes made to code or templates; separating etc and the rest of Koha into submodules would make it harder to prepare patches and manage topic branches. My idea for that is that tracking addition of zebra indexes and all that stuff would be considerably eased if there was an initial etc repository
Le 10/11/2010 19:10, Galen Charlton a écrit : that installer clone into /home/koha/kohadev Then makes the updates on strings and commit. All the index to fit customized library need could be tracked. We create a git repository for all the installations, but when you create the repository from the processed files, it looses synchronization with all the common indexes which could be required for Koha when one adds some other minor feature or fixes a bug in the indexer. An other reason is that etc doesnot vary much. But when it varies, when you upgrade users should be aware that they might loose their custom indexes. I wanted to make next upgrades smoother for libraries. I am with you when you propose to use different directories. This would bring to : etc |- koha-conf.xml or koha-conf.yml i.e. ONLY Koha common configuration (database access and so on. No more zebra stuff in that. ) |- authentication ||- LDAP ||- CAS |-webserver ||- apache2 ||- nginx |- searchengine ||-zebradb ||-solr ||-pazpar2 Are you with me ? But then, when one chooses one type of webserver, one type of authentication, one type of searchengine, he would use only a few of all the installation files (which could become quite a forest). I wanted the structure for etc simpler so that sysadmins would not be overwhelmed by big picture.
0 for moving the PO files to a separate Git project. The size of the repository doesn't really strike me as a big deal; the Git protocol is pretty efficient. That said, while I don't see a great deal of benefit to splitting the translations off into a separate repository, I don't see much harm either. Well it is quite striking when you get 249Mo to dl when doing a git clone. :) (mainly because any change you commit on po files is storing a new instance of this file) ADSL is coping well with that... But there are still some places in the world which donot have access to wide bandwidth.
Regards. -- Henri-Damien LAURENT
Hi, On Sun, Nov 14, 2010 at 11:48 AM, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
My idea for that is that tracking addition of zebra indexes and all that stuff would be considerably eased if there was an initial etc repository that installer clone into /home/koha/kohadev
Since adding a Zebra index is sometimes paired with an update to template files, splitting the configuration into a separate repository would actually make it more difficult to track Zebra updates. It gets worse if you consider changes such as adding or modifying an authentication module, for which the default configuration changes are almost invariably associated with code changes. What you're proposing would make it more difficult to cleanly manage Koha development that touches the default configuration files; among other problems, Git's submodule support is such that it would require that users and developers would have to do more than a simple git pull or git fetch/rebase in order to ensure that they've fetched updates both to Koha and to its configuration files.
We create a git repository for all the installations, but when you create the repository from the processed files, it looses synchronization with all the common indexes which could be required for Koha when one adds some other minor feature or fixes a bug in the indexer.
Since it sounds like you're using dev-mode deployments for your customers, you (i.e., BibLibre, not necessarily you personally) could just as readily keep track of local customizations in the git clone for each installation you run, then do a make update_zebra_conf. I'm sure that there are a variety of ways to script that, and perhaps some patches to Makefile.PL would help you better than splitting out etc into a submodule.
An other reason is that etc doesnot vary much. But when it varies, when you upgrade users should be aware that they might loose their custom indexes. I wanted to make next upgrades smoother for libraries.
My suggestion immediately above should help you, but I also want to point out that make upgrade *does* create backups of any files it touches, so local changes are preserved that way. I am not suggesting that there aren't things that could be done to make things easier for anybody who runs a lot of dev-mode Koha installations, but sticking etc off into a submodule is not the solution.
I am with you when you propose to use different directories. This would bring to : etc |- koha-conf.xml or koha-conf.yml i.e. ONLY Koha common configuration (database access and so on. No more zebra stuff in that. )
I would support that. It's mostly a historical accident that koha-conf.xml currently serves as both the main configuration file for Koha and the top-level config file for zebrasrv, but there's no reason why those two configuration functions couldn't be placed in separate files.
|- authentication ||- LDAP ||- CAS |-webserver ||- apache2 ||- nginx |- searchengine ||-zebradb ||-solr ||-pazpar2
I'm OK with splitting the configuration files.
But then, when one chooses one type of webserver, one type of authentication, one type of searchengine, he would use only a few of all the installation files (which could become quite a forest). I wanted the structure for etc simpler so that sysadmins would not be overwhelmed by big picture.
But all of the options have to exist *somewhere*, and it would be simpler to manage the development of the various options if the configuration files and directories were all laid out directly in the Git repository, not relegated to topic branches. Furthermore, if all of the possible configuration files are available in a production installation, it would be easier for a sysadmin to (say) switch from Apache to nginx. In other words, I think it is better to organize the configuration files well (and document them!) than to effectively atomize the management of them during Koha development by having permanent topic branches for various configuration modes.
0 for moving the PO files to a separate Git project. The size of the repository doesn't really strike me as a big deal; the Git protocol is pretty efficient. That said, while I don't see a great deal of benefit to splitting the translations off into a separate repository, I don't see much harm either.
As I said, I don't have an strong opinion either way -- that's what the 0 means -- but I do think there are a couple misconceptions to clear up:
Well it is quite striking when you get 249Mo to dl when doing a git clone. :)
Not sure where you're getting 249M from -- it's more like 151 MiB when I measured today.
(mainly because any change you commit on po files is storing a new instance of this file)
Actually, no, it doesn't. Git is better designed than that; generally what it would do if you commit a change to a PO file is store just the delta. git gc is run on the public repo every week. When you push or pull to a repository, Git transfers just compressed deltas.
ADSL is coping well with that... But there are still some places in the world which donot have access to wide bandwidth.
True. But a Git clone (of the public repo) is a once-and-done operation. Anybody installing Koha for production use could use the tarball or (even better) the Debian package. Particularly because of the Debian package, we're getting past the point where dev mode would be recommend for use by single-library production installations. I've been doing some measurements. A PO-only repository would be about 50M in size, and creating such a thing is the easy part. But if we move misc/translator/po to a separate repository, we would have to also remove that directory from the main repository in order to realize the repository size savings motivating your proposal - a 'git rm misc/translator/po' wouldn't reduce the size of the repo. My test run is not quite finished yet (it takes a long time for git-filter-branch to handle almost 13,000 commits), but even assuming that 50M could be pared from the main repository, actually doing that would come at a significant cost: every commit would be rewritten by the git-filter-branch operation. Rewriting history like that could mean that every single person who clones against the public repo could have to deal with forced branch updates, to say nothing of invalidating all of the release tags. That prospect doesn't hearten me. I'll report back once my test finishes. Regards, Galen -- Galen Charlton gmcharlt@gmail.com
Le 14/11/2010 22:51, Galen Charlton a écrit :
Hi,
On Sun, Nov 14, 2010 at 11:48 AM, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
My idea for that is that tracking addition of zebra indexes and all that stuff would be considerably eased if there was an initial etc repository that installer clone into /home/koha/kohadev
Since adding a Zebra index is sometimes paired with an update to template files, splitting the configuration into a separate repository would actually make it more difficult to track Zebra updates. I consider this as a design flaw, could be overcome with work and
collaboration on a better abstraction for lists of indexes. (It is already work in progress for Solr. Could be quite long to implement with zebra, but achievable, but it would only be a matter of parsing the ccl.properties file and proposing an interface for better management)
It gets worse if you consider changes such as adding or modifying an authentication module, for which the default configuration changes are almost invariably associated with code changes. Same for authentication. I consider this as a design flaw. We should split authentication and identification. And manage identification in a modular way... So that administrators would just have to edit configuration files in order to make correct mappings, and not dive into the code, change and commit (if they know git enough...)
What you're proposing would make it more difficult to cleanly manage Koha development that touches the default configuration files; among other problems, Git's submodule support is such that it would require that users and developers would have to do more than a simple git pull or git fetch/rebase in order to ensure that they've fetched updates both to Koha and to its configuration files. Well changing to git was already making things kind of a little bit hard for non git users. Change was good and now everybody is using it (at least we are even proposing ppl) Clear and documented information would overcome this fact.
We create a git repository for all the installations, but when you create the repository from the processed files, it looses synchronization with all the common indexes which could be required for Koha when one adds some other minor feature or fixes a bug in the indexer.
Since it sounds like you're using dev-mode deployments for your customers, you (i.e., BibLibre, not necessarily you personally) could just as readily keep track of local customizations in the git clone for each installation you run, then do a make update_zebra_conf. I'm sure that there are a variety of ways to script that, and perhaps some patches to Makefile.PL would help you better than splitting out etc into a submodule.
An other reason is that etc doesnot vary much. But when it varies, when you upgrade users should be aware that they might loose their custom indexes. I wanted to make next upgrades smoother for libraries.
My suggestion immediately above should help you, but I also want to point out that make upgrade *does* create backups of any files it touches, so local changes are preserved that way.
Nice.
I am not suggesting that there aren't things that could be done to make things easier for anybody who runs a lot of dev-mode Koha installations, but sticking etc off into a submodule is not the solution. Well it was a proposal.
I am with you when you propose to use different directories. This would bring to : etc
That organisation was the big picture, what I would envision for long term. I didnot mean that would have to be done right away.
|- koha-conf.xml or koha-conf.yml i.e. ONLY Koha common configuration (database access and so on. No more zebra stuff in that. )
I would support that. It's mostly a historical accident that koha-conf.xml currently serves as both the main configuration file for Koha and the top-level config file for zebrasrv, but there's no reason why those two configuration functions couldn't be placed in separate files.
|- authentication ||- LDAP ||- CAS |-webserver ||- apache2 ||- nginx |- searchengine ||-zebradb ||-solr ||-pazpar2
I'm OK with splitting the configuration files.
But then, when one chooses one type of webserver, one type of authentication, one type of searchengine, he would use only a few of all the installation files (which could become quite a forest). I wanted the structure for etc simpler so that sysadmins would not be overwhelmed by big picture.
But all of the options have to exist *somewhere*, and it would be simpler to manage the development of the various options if the configuration files and directories were all laid out directly in the Git repository, not relegated to topic branches. Furthermore, if all of the possible configuration files are available in a production installation, it would be easier for a sysadmin to (say) switch from Apache to nginx. Why not... But then, you would have to make dependencies quite
thorough... Or optional. debconf would surely be quite handy for that. But would need some clarifications.
In other words, I think it is better to organize the configuration files well (and document them!) than to effectively atomize the management of them during Koha development by having permanent topic branches for various configuration modes.
0 for moving the PO files to a separate Git project. The size of the repository doesn't really strike me as a big deal; the Git protocol is pretty efficient. That said, while I don't see a great deal of benefit to splitting the translations off into a separate repository, I don't see much harm either.
As I said, I don't have an strong opinion either way -- that's what the 0 means -- but I do think there are a couple misconceptions to clear up:
Well it is quite striking when you get 249Mo to dl when doing a git clone. :)
Not sure where you're getting 249M from -- it's more like 151 MiB when I measured today.
(mainly because any change you commit on po files is storing a new instance of this file)
Actually, no, it doesn't. Git is better designed than that; generally what it would do if you commit a change to a PO file is store just the delta. git gc is run on the public repo every week. When you push or pull to a repository, Git transfers just compressed deltas. From my understanding of Scott Chacon (a git developer) book and git
talk, git stores the files in its database, since it is a DAG oriented SCM. It may send the diffs, but actualy stores the files.
ADSL is coping well with that... But there are still some places in the world which donot have access to wide bandwidth.
True. But a Git clone (of the public repo) is a once-and-done operation. Anybody installing Koha for production use could use the tarball or (even better) the Debian package. Particularly because of the Debian package, we're getting past the point where dev mode would be recommend for use by single-library production installations.
I've been doing some measurements. A PO-only repository would be about 50M in size, and creating such a thing is the easy part. But if we move misc/translator/po to a separate repository, we would have to also remove that directory from the main repository in order to realize the repository size savings motivating your proposal - a 'git rm misc/translator/po' wouldn't reduce the size of the repo. My test run is not quite finished yet (it takes a long time for git-filter-branch to handle almost 13,000 commits), but even assuming that 50M could be pared from the main repository, actually doing that would come at a significant cost: every commit would be rewritten by the git-filter-branch operation. Rewriting history like that could mean that every single person who clones against the public repo could have to deal with forced branch updates, to say nothing of invalidating all of the release tags.
I don't think all the release tags would be broken. And it would allow to release localisation at a different pace... When there is a need.
That prospect doesn't hearten me. I'll report back once my test finishes.
Thanks for your update.
Regards,
Galen
Regards. -- Henri-Damien LAURENT
I like the idea of separating out the .po files if for no other reason than it expediting "grep -r". Reducing the overall repo footprint is also helpful. It doesn't matter much for a single instance, but it does add up when you have dozens of them. Clay On Wed, Nov 10, 2010 at 8:11 AM, LAURENT Henri-Damien < henridamien.laurent@biblibre.com> wrote:
Hi I propose to get etc and translator as external git repositories. Why ?
It would allow to have po files in their own repository and limit the weight of Koha git archives. For etc, it would allow to change the installer in order to keep track of local changes for every person and therefore be able to synchronize with community version properly.
HOW ? operation is simple : git filter-branch --subdirectory-filter misc/translator -- --all and then git push on an external repository. for instance etc.git and translator.git on git.koha-community This would allow to have more than one branch for etc and for instance create and etc/apache etc/nginx etc/plack and so on...
WHEN ? when you think operation should be done, if you think it could be interesting.
and then we could have git submodules in the koha.git repository in order to keep in synch and synchronise versions when we want. http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html
It is just a proposition. -- Henri-Damien LAURENT _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
participants (7)
-
Chris Nighswonger -
Clay Fouts -
Frédéric Demians -
Galen Charlton -
LAURENT Henri-Damien -
LAURENT Henri-Damien -
Robin Sheat