git repo size (restarting a discussion)
Hello koha-devel, Some months ago, Frederic started an interesting thread about translations and their consequences on the git repository size. The wiki page speaking of this is on http://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking We have decided nothing and only a few of us have expressed their opinion. Now, there is a big patch from Ukraine, that add another 2MB to Koha repo size (http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6434) Frederic proposes 3 scenarios: 1- Keep history -- Create two (or more) Koha sub-projects, one for Koha code koha-core, and one for Koha .po files koha-i18n, and keep the whole history in both of the repositories. 2- Keep history as history -- We froze the current Git repository in which Koha history will live for eternity as a koha-legacy repo, and initialize two new clean and fresh repositories. 3- Keep-loose history -- Keep the history in koha-core, start fresh for koha-l8n. The gain is impressive (see: http://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking#Evaluation) When Frederic wrote the page, the repository was 177MB large. Now, it is 355 ! I think we can't continue that way, and should decide something before 3.8 is released (in 3 weeks) Everyone is welcomed to read the wiki page, express his opinion, I propose to have a vote on the next IRC meeting. Frederic = isn't there another option ? = a huge "git rebase -i" where we would discard all commits that are related to .po files (the list being found with git log --oneline misc/translator/po (i haven't checked it work, just rising the idea) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Paul Poulain <paul.poulain@biblibre.com>
Some months ago, Frederic started an interesting thread about translations and their consequences on the git repository size. The wiki page speaking of this is on http://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking
I think part of the problem is that doing a good shallow clone (which would be fine for interacting with git bz and by email) is difficult because of the number of branches that are in the repository. I think there's something like 1500 branches. When you use git clone --depth 0, it tracks each branch head, so those 1500 branches mean that almost everything is downloaded. If you do instead: mkdir shallow cd shallow git init git remote add origin git://git.koha-community.org/koha.git git pull --depth 0 -n origin master then you get a 113M .git instead of one about 300M. I still think there's probably value in rebooting the repository (I have a slight preference for option 2 or 3, but no strong feeling), but please next time can we delete branches once they are merged? Or is there a drawback to deleting merged branches that I don't see? Thanks, -- MJ Ray (slef), member of www.software.coop, a for-more-than-profit co-op. http://koha-community.org supporter, web and library systems developer. In My Opinion Only: see http://mjr.towers.org.uk/email.html Available for hire (including development) at http://www.software.coop/
When you use git clone --depth 0, it tracks each branch head, so those 1500 branches mean that almost everything is downloaded.
It's rather a matter of unused git object pruning. With: git clone --mirror git://git.koha-community.org/koha You get a repo whose size is 280M. Then with: git gc --aggressive --prune=now You drop repo size to 138M. I suggest to do some maintenance on community repository. -- Frédéric DEMIANS http://www.tamil.fr/u/fdemians.html
Le 05/04/2012 08:08, Frédéric Demians a écrit :
git gc --aggressive --prune=now
You drop repo size to 138M.
Yikes: I just tried twice with my laptop -so locally- and get a nasty error after 10mn taking 100% of my 4 cores = 11:01 ~/koha.dev/koha-community (master $% u=)$ git gc --aggressive --prune=now Counting objects: 296296, done. Delta compression using up to 4 threads. fatal: Out of memory, malloc failed (tried to allocate 4479308 bytes) error: failed to run repack (I have no access to the git repo itself, I think it's something galen or chris only can do) PS: I think doing this reduction would be a great thing, (and probably enough) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Paul Poulain <paul.poulain@biblibre.com>
Yikes: I just tried twice with my laptop -so locally- and get a nasty error after 10mn taking 100% of my 4 cores = [...] fatal: Out of memory, malloc failed (tried to allocate 4479308 bytes) error: failed to run repack
Did it leave the repository in a usable state?
(I have no access to the git repo itself, I think it's something galen or chris only can do)
PS: I think doing this reduction would be a great thing, (and probably enough)
I've raised it with Chris on IRC. http://irc.koha-community.org/irclog/koha/2012-04-05#i_940627 Hope that helps, -- MJ Ray (slef), member of www.software.coop, a for-more-than-profit co-op. http://koha-community.org supporter, web and library systems developer. In My Opinion Only: see http://mjr.towers.org.uk/email.html Available for hire (including development) at http://www.software.coop/
Le 05/04/2012 11:36, MJ Ray a écrit :
Paul Poulain <paul.poulain@biblibre.com>
Yikes: I just tried twice with my laptop -so locally- and get a nasty error after 10mn taking 100% of my 4 cores = [...] fatal: Out of memory, malloc failed (tried to allocate 4479308 bytes) error: failed to run repack
Did it leave the repository in a usable state? Seems yes (git status, git branch, git checkout, git checkout -b, git log seems to work fine) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
On 2012-04-5, at 11:25 PM, Paul Poulain wrote:
Le 05/04/2012 11:36, MJ Ray a écrit :
Paul Poulain <paul.poulain@biblibre.com>
Yikes: I just tried twice with my laptop -so locally- and get a nasty error after 10mn taking 100% of my 4 cores = [...] fatal: Out of memory, malloc failed (tried to allocate 4479308 bytes) error: failed to run repack
i had the same problem too. i had a google, and found some good advice http://stackoverflow.com/questions/9561835/git-push-suboptimal-pack-out-of-m... http://stackoverflow.com/questions/7362709/git-pull-fails-with-bad-pack-head... i added this to my ~/.gitconfig file, and it fixed my problem :) [pack] threads = 1 these also may help..... git config --global pack.windowMemory "100m" git config --global pack.SizeLimit "100m" git config --global pack.threads "1"
On 2012-04-6, at 4:29 PM, Mason James wrote:
On 2012-04-5, at 11:25 PM, Paul Poulain wrote:
Le 05/04/2012 11:36, MJ Ray a écrit :
Paul Poulain <paul.poulain@biblibre.com>
Yikes: I just tried twice with my laptop -so locally- and get a nasty error after 10mn taking 100% of my 4 cores = [...] fatal: Out of memory, malloc failed (tried to allocate 4479308 bytes) error: failed to run repack
i had the same problem too. i had a google, and found some good advice
http://stackoverflow.com/questions/9561835/git-push-suboptimal-pack-out-of-m... http://stackoverflow.com/questions/7362709/git-pull-fails-with-bad-pack-head...
aaah, perhaps this is a better option -> https://github.com/hbons/SparkleShare/issues/519 leave git multi-threading on, but force some memory limits, with a ~/.gitconfig file like this... [core] packedGitLimit = 128m packedGitWindowSize = 128m [pack] deltaCacheSize = 128m packSizeLimit = 128m windowMemory = 128m
Hi, On Apr 5, 2012, at 2:08 AM, Frédéric Demians wrote:
Then with:
git gc --aggressive --prune=now
You drop repo size to 138M.
I suggest to do some maintenance on community repository.
I'm running a test of this now on a copy of the main Koha repository on git.koha-community.org. Assuming I don't run into any problems, I can schedule a run for the main repository this weekend. To be safe, I would plan on disabling pushing to the repository while the garbage collection runs (it looks like it might take a couple hours), so I'd like folks with push access to confirm suitable windows of time. I'll announce on #koha when I get started and finish. Regards, Galen -- Galen Charlton Director of Support and Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
Le 20/04/2012 17:51, Galen Charlton a écrit :
Hi, Hi Galen,
I'm running a test of this now on a copy of the main Koha repository on git.koha-community.org. Assuming I don't run into any problems, I can schedule a run for the main repository this weekend. To be safe, I would plan on disabling pushing to the repository while the garbage collection runs (it looks like it might take a couple hours), so I'd like folks with push access to confirm suitable windows of time. I'll announce on #koha when I get started and finish. I'll release Koha 3.8.0 on monday morning
Maybe it's wise to wait for after this date ? I won't push anything to the main repo after releasing for one week, because i'll be AFK ! -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
participants (5)
-
Frédéric Demians -
Galen Charlton -
Mason James -
MJ Ray -
Paul Poulain