How about a different way of handling database updates
Database updates using the updatedatabase.pl script. I'm thinking there could be a better way, both for developers and the release maintainers. The problem is that we have a sequential number controlling the updates. This is working now, but I think there could be a better way. The current method is simple, it has that benefit. On the other hand it makes things more complex for the release maintainer to make sure the sequence is right. It's also more complex for the developers to coordinate who will use what number. My idea is to use a string for the update instead of a number. It will mean adding a table to the database to track update strings that have been applied. This way each developer, or company, uses their own string to distinguish updates. Also, in order to keep the table from being huge, the release maintainer will occasionally announce a database revision. We will keep the number to effectively federate the updates. Within each revision there will be any number of update strings. With each new revision it is assumed that the updates from the previous version are applied, so the database table is emptied. There will be some added complexity when when there is a revision. The developers who are working on update code will have to move their in the updatedatabase.pl file into the section for the new revision. The added benefit of this method is that developers, and companies, can have a database update easily without worrying about it getting into the official code repositories. The change is kept in the developers own branch, with it's unique update string, while others that are accepted into the official code are also there. Does that make sense? Have I described it well enough? Is anyone interested in this idea? I've been thinking about it, and the changes to the database update process should be minimal. There will be significant changes, but not many of them. There will have to be a sub added to check if an update string is in the database. And a sub to add the update string to the tracking table in the database. And the revision update sub will have to be expanded to empty that table. I think that would be the extent of the changes. Comments please. -- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
I like this idea. I've had a number of problems with the current system. If I understand correctly, we would need a database table to track updates. It sounds like we would only need a single column that would contain the title of the update, say "BugFix2235" for example. The update script would then check the db_revisions table to see if "BugFix2235" is in there. If not, it would execute the update and add "BugFix2235" to the db_revisions table. We could also continue updating the Version pref like we have been.
Also, in order to keep the table from being huge, the release maintainer will occasionally announce a database revision. We will keep the number to effectively federate the updates. Within each revision there will be any number of update strings. With each new revision it is assumed that the updates from the previous version are applied, so the database table is emptied.
This part does not sound necessary to myself. We could just keep the existing updatedatabase.pl and just add a new sub to handle the new system, and continue on with it. Kyle http://www.kylehall.info Information Technology Crawford County Federated Library System ( http://www.ccfls.org ) On Wed, Jan 27, 2010 at 6:28 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
Database updates using the updatedatabase.pl script. I'm thinking there could be a better way, both for developers and the release maintainers.
The problem is that we have a sequential number controlling the updates. This is working now, but I think there could be a better way. The current method is simple, it has that benefit. On the other hand it makes things more complex for the release maintainer to make sure the sequence is right. It's also more complex for the developers to coordinate who will use what number.
My idea is to use a string for the update instead of a number. It will mean adding a table to the database to track update strings that have been applied. This way each developer, or company, uses their own string to distinguish updates.
Also, in order to keep the table from being huge, the release maintainer will occasionally announce a database revision. We will keep the number to effectively federate the updates. Within each revision there will be any number of update strings. With each new revision it is assumed that the updates from the previous version are applied, so the database table is emptied.
There will be some added complexity when when there is a revision. The developers who are working on update code will have to move their in the updatedatabase.pl file into the section for the new revision.
The added benefit of this method is that developers, and companies, can have a database update easily without worrying about it getting into the official code repositories. The change is kept in the developers own branch, with it's unique update string, while others that are accepted into the official code are also there.
Does that make sense? Have I described it well enough? Is anyone interested in this idea?
I've been thinking about it, and the changes to the database update process should be minimal. There will be significant changes, but not many of them. There will have to be a sub added to check if an update string is in the database. And a sub to add the update string to the tracking table in the database. And the revision update sub will have to be expanded to empty that table. I think that would be the extent of the changes.
Comments please.
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Clearing the tracking table isn't necessary. I'm thinking 50 versions down the road when the tracking table has some 500 strings. This makes it a little harder to ensure a unique string for the update. And I would rather the table stay small to be quicker to index and to take up less space on the database server. Since it's just the one string, which has to be unique anyway, this is a very insignificant concern. My thoughts on the title of the update though are a little different from your example. I'll explain here so you all know what I'm thinking. I think an example would best illustrate. If it were the title of one of my updates it might be 'mdhafen.001' or 'wcsd.3_0_0.001'. Or maybe even 'wcsd.mdhafen.001'. I'm just beginning to think of the details of the title scheme. I think that third example is what I'll end up with. For Kyle perhaps the title would be 'ccfls.kyle_m_hall.001'. So that's what I'm thinking; that should make it easy to have unique update titles. Comments? On Thu, 2010-01-28 at 07:51 -0500, Kyle Hall wrote:
I like this idea. I've had a number of problems with the current system. If I understand correctly, we would need a database table to track updates. It sounds like we would only need a single column that would contain the title of the update, say "BugFix2235" for example. The update script would then check the db_revisions table to see if "BugFix2235" is in there. If not, it would execute the update and add "BugFix2235" to the db_revisions table. We could also continue updating the Version pref like we have been.
Also, in order to keep the table from being huge, the release maintainer will occasionally announce a database revision. We will keep the number to effectively federate the updates. Within each revision there will be any number of update strings. With each new revision it is assumed that the updates from the previous version are applied, so the database table is emptied.
This part does not sound necessary to myself. We could just keep the existing updatedatabase.pl and just add a new sub to handle the new system, and continue on with it.
Kyle
http://www.kylehall.info Information Technology Crawford County Federated Library System ( http://www.ccfls.org )
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
The naming convention sounds very reasonable. Very java import styled. Very good idea. Now the big question is, can we get any traction for such a big change? Is there anyone else willing to comment on the possbility of this idea? Kyle http://www.kylehall.info Information Technology Crawford County Federated Library System ( http://www.ccfls.org ) On Thu, Jan 28, 2010 at 10:46 AM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
Clearing the tracking table isn't necessary. I'm thinking 50 versions down the road when the tracking table has some 500 strings. This makes it a little harder to ensure a unique string for the update. And I would rather the table stay small to be quicker to index and to take up less space on the database server. Since it's just the one string, which has to be unique anyway, this is a very insignificant concern.
My thoughts on the title of the update though are a little different from your example. I'll explain here so you all know what I'm thinking. I think an example would best illustrate. If it were the title of one of my updates it might be 'mdhafen.001' or 'wcsd.3_0_0.001'. Or maybe even 'wcsd.mdhafen.001'. I'm just beginning to think of the details of the title scheme. I think that third example is what I'll end up with. For Kyle perhaps the title would be 'ccfls.kyle_m_hall.001'. So that's what I'm thinking; that should make it easy to have unique update titles.
Comments?
On Thu, 2010-01-28 at 07:51 -0500, Kyle Hall wrote:
I like this idea. I've had a number of problems with the current system. If I understand correctly, we would need a database table to track updates. It sounds like we would only need a single column that would contain the title of the update, say "BugFix2235" for example. The update script would then check the db_revisions table to see if "BugFix2235" is in there. If not, it would execute the update and add "BugFix2235" to the db_revisions table. We could also continue updating the Version pref like we have been.
Also, in order to keep the table from being huge, the release maintainer will occasionally announce a database revision. We will keep the number to effectively federate the updates. Within each revision there will be any number of update strings. With each new revision it is assumed that the updates from the previous version are applied, so the database table is emptied.
This part does not sound necessary to myself. We could just keep the existing updatedatabase.pl and just add a new sub to handle the new system, and continue on with it.
Kyle
http://www.kylehall.info Information Technology Crawford County Federated Library System ( http://www.ccfls.org )
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
Kyle Hall a écrit :
The naming convention sounds very reasonable. Very java import styled. Very good idea.
Now the big question is, can we get any traction for such a big change? Is there anyone else willing to comment on the possbility of this idea?
create a syspref (with TEXT, we have a lot of space). Separate each patch applied with a comma : BibLibre-1,mhafenA,someoneElse,anotherone,BibLibre-2, ... need to check that a patch has been applied ? just =~ /BibLibre-1,/ if not applied, apply it, and add BibLibre-1, at the end of applied patches. The order doesn't matter. (well, in fact, the order may matter, and that may be the limit of this idea...) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
create a syspref (with TEXT, we have a lot of space). Separate each patch applied with a comma :
BibLibre-1,mhafenA,someoneElse,anotherone,BibLibre-2, ...
need to check that a patch has been applied ? just =~ /BibLibre-1,/ if not applied, apply it, and add BibLibre-1, at the end of applied patches. The order doesn't matter.
(well, in fact, the order may matter, and that may be the limit of this idea...)
An interesting idea, but wouldn't that cause the db updater to slow down as the syspref grows in size? Kyle
On Thu, Jan 28, 2010 at 1:04 PM, Kyle Hall <kyle.m.hall@gmail.com> wrote:
create a syspref (with TEXT, we have a lot of space). Separate each patch applied with a comma :
BibLibre-1,mhafenA,someoneElse,anotherone,BibLibre-2, ...
need to check that a patch has been applied ? just =~ /BibLibre-1,/ if not applied, apply it, and add BibLibre-1, at the end of applied patches. The order doesn't matter.
(well, in fact, the order may matter, and that may be the limit of this idea...)
An interesting idea, but wouldn't that cause the db updater to slow down as the syspref grows in size?
Probably not as regexp's are one of perl's strong points. Chris
Update order. I hadn't thought of that. That could be a big deal. I'm already committed to writing it if it does get traction. I have a couple database updates in my own branch that couldn't be in the the updatedatabase.pl obviously. If this does get traction and we can figure out the patch order thing I'll want this a lot. The patch order may not be that big a deal. There is an implicit patch order enforced by the patches location in the updatedatabase.pl file. That might be good enough. On Thu, 2010-01-28 at 18:29 +0100, Paul Poulain wrote:
Kyle Hall a écrit :
The naming convention sounds very reasonable. Very java import styled. Very good idea.
Now the big question is, can we get any traction for such a big change? Is there anyone else willing to comment on the possbility of this idea?
create a syspref (with TEXT, we have a lot of space). Separate each patch applied with a comma :
BibLibre-1,mhafenA,someoneElse,anotherone,BibLibre-2, ...
need to check that a patch has been applied ? just =~ /BibLibre-1,/ if not applied, apply it, and add BibLibre-1, at the end of applied patches. The order doesn't matter.
(well, in fact, the order may matter, and that may be the limit of this idea...)
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
I would have to agree the patch order probably won't be an issue, as each patch will still be added in the chronological order that it was committed ( assuming every always adds to the end of the file as usual ). Kyle http://www.kylehall.info Information Technology Crawford County Federated Library System ( http://www.ccfls.org ) On Thu, Jan 28, 2010 at 4:07 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote:
Update order. I hadn't thought of that. That could be a big deal.
I'm already committed to writing it if it does get traction. I have a couple database updates in my own branch that couldn't be in the the updatedatabase.pl obviously. If this does get traction and we can figure out the patch order thing I'll want this a lot.
The patch order may not be that big a deal. There is an implicit patch order enforced by the patches location in the updatedatabase.pl file. That might be good enough.
On Thu, 2010-01-28 at 18:29 +0100, Paul Poulain wrote:
Kyle Hall a écrit :
The naming convention sounds very reasonable. Very java import styled. Very good idea.
Now the big question is, can we get any traction for such a big change? Is there anyone else willing to comment on the possbility of this idea?
create a syspref (with TEXT, we have a lot of space). Separate each patch applied with a comma :
BibLibre-1,mhafenA,someoneElse,anotherone,BibLibre-2, ...
need to check that a patch has been applied ? just =~ /BibLibre-1,/ if not applied, apply it, and add BibLibre-1, at the end of applied patches. The order doesn't matter.
(well, in fact, the order may matter, and that may be the limit of this idea...)
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA
for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
Michael Hafen a écrit :
Update order. I hadn't thought of that. That could be a big deal.
Well it could be important at some point : you cannot update a field which doesnot exist. And so on....
I'm already committed to writing it if it does get traction. I have a couple database updates in my own branch that couldn't be in the the updatedatabase.pl obviously. If this does get traction and we can figure out the patch order thing I'll want this a lot.
Another thing that I like about your idea is that it could point to atomic change files... Easily test-able, easily indentified in a hierarchy of folders. BibLibre-Lyon-1 could make call for instance to biblibre/Lyon/1.pl ... Therefore, we would not end up with having merge conflicts. Maybe we could add to pl file a do and undo function so that... we can do and undo those updates. (But this is just day dreaming, just before going to bed.) I can't wait for your proposal :P My two cents though ;) -- Henri-Damien LAURENT
The patch order may not be that big a deal. There is an implicit patch order enforced by the patches location in the updatedatabase.pl file.
That might be good enough.
On Thu, 2010-01-28 at 18:29 +0100, Paul Poulain wrote:
Kyle Hall a écrit :
The naming convention sounds very reasonable. Very java import styled. Very good idea.
Now the big question is, can we get any traction for such a big change? Is there anyone else willing to comment on the possbility of this idea?
create a syspref (with TEXT, we have a lot of space). Separate each patch applied with a comma :
BibLibre-1,mhafenA,someoneElse,anotherone,BibLibre-2, ...
need to check that a patch has been applied ? just =~ /BibLibre-1,/ if not applied, apply it, and add BibLibre-1, at the end of applied patches. The order doesn't matter.
(well, in fact, the order may matter, and that may be the limit of this idea...)
Calling out to atomic files would be a good idea. It only partially alleviates merge conflicts though, because the reference to the file still have to appear in updatedatabase.pl. You would end up missing an entire update rather than potentially messing up part of an update with a merge conflict. Missing the update is probably the better outcome though, and the updatedatabase.pl file will be much smaller that way. So having atomic update files, referenced from updatedatabase.pl, for large updates (several lines of code) would be part of my proposal. As for having do() and undo() in the included file... It's a nice idea, but in practice un-deleting a column can be difficult. Imagine trying to reverse the database changes from the labels rewrite or the acquisitions rewrite. I think that standard practice for updating is to backup the database first. Then restore from the backup if there is a problem. This practice falls flat though if you don't notice the problem right away. In which case having an undo() would be preferable. But it would be totally dependent on being able to pull the information from somewhere to fill the un-deleted column. Really the reason I don't like the idea is because I'm lazy. That practically doubles the work it takes to change the database. Comments? On Thu, 2010-01-28 at 22:46 +0100, LAURENT Henri-Damien wrote:
Michael Hafen a écrit :
Update order. I hadn't thought of that. That could be a big deal.
Well it could be important at some point : you cannot update a field which doesnot exist. And so on....
I'm already committed to writing it if it does get traction. I have a couple database updates in my own branch that couldn't be in the the updatedatabase.pl obviously. If this does get traction and we can figure out the patch order thing I'll want this a lot.
Another thing that I like about your idea is that it could point to atomic change files... Easily test-able, easily indentified in a hierarchy of folders. BibLibre-Lyon-1 could make call for instance to biblibre/Lyon/1.pl ... Therefore, we would not end up with having merge conflicts.
Maybe we could add to pl file a do and undo function so that... we can do and undo those updates. (But this is just day dreaming, just before going to bed.) I can't wait for your proposal :P
My two cents though ;)
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
Michael Hafen a écrit :
Calling out to atomic files would be a good idea. It only partially alleviates merge conflicts though, because the reference to the file still have to appear in updatedatabase.pl. You would end up missing an entire update rather than potentially messing up part of an update with a merge conflict. Missing the update is probably the better outcome though, and the updatedatabase.pl file will be much smaller that way.
So having atomic update files, referenced from updatedatabase.pl, for large updates (several lines of code) would be part of my proposal.
Could be in fact a loop calling update files while $data->{active}{ my $filecontent=File::Slurp(getfilename($data->{dbrev_name})) eval {$filecontent}} and not including them in the update file. Imagine we had a column activate in your table where an active update would HAVE to be done. We would be able to "check" that it has been done. I think that we should not have to do the loop over and over again at each page loading. But have a maintenance or an about screen where you can check against the updates applied and not applied.
As for having do() and undo() in the included file... It's a nice idea, but in practice un-deleting a column can be difficult. Imagine trying to reverse the database changes from the labels rewrite or the acquisitions rewrite.
I think that standard practice for updating is to backup the database first. Then restore from the backup if there is a problem. This practice falls flat though if you don't notice the problem right away. In which case having an undo() would be preferable. But it would be totally dependent on being able to pull the information from somewhere to fill the un-deleted column.
Really the reason I don't like the idea is because I'm lazy. That practically doubles the work it takes to change the database.
Comments?
I am lazy too... not only, busy also. And I would like the process to be as sure as possible so that maintenance is less time consuming. But this was day dreaming and brain dumping. -- Henri-Damien LAURENT BibLibre
Those kind of things can be tricky and messy. Something like a dependency graph would be required, which is all but easy to implement. It has something to do with how patches are applied with git: order matters. 'git log --graph --pretty-oneline' displays s a representation of this order. Git manages pretty well its infamous 'conflicts' when merging branches, applying patches. Not sure it's possible to do that properly with DB updates. My opinion is that the current system is not that bad. The dependency graph is linear, date ordered by version number last digits. It just becomes crazy when updates are ported from one branch to the other, HEAD to maintenance for example. And there is (am I wrong?) to claim 'communitly' such a number: http://wiki.koha.org/doku.php?id=en:development:dbrevs:start It should be used systematically. Extracting updates code from updatabase.pl script and putting it into a directory containing files named by version number could be a simplification: easier for RM to rename those files if required, not sure. It remains kohastructure.sql... -- Frédéric
I think this idea has snowballed into something way more complex than the original idea, which I still prefer.
My opinion is that the current system is not that bad. The dependency graph is linear, date ordered by version number last digits. It just becomes crazy when updates are ported from one branch to the other, HEAD to maintenance for example. And there is (am I wrong?) to claim 'communitly' such a number:
The problem with the wiki page for db number claiming is it doesn't seem to work, at least it hasn't worked for me in the past. It assumes that the features will be committed in the order that the db numbers are claimed, which I don't believe is true. In that case, if the wiki page were enforced, then db versions would be committed out of order, and the updater would not work correctly. I just don't believe the wiki page is effective at all. After having a number of db version numbers stepped on, I stopped using it. Kyle http://www.kylehall.info Information Technology Crawford County Federated Library System ( http://www.ccfls.org ) On Fri, Jan 29, 2010 at 2:24 AM, Frederic Demians <frederic@tamil.fr> wrote:
Those kind of things can be tricky and messy. Something like a dependency graph would be required, which is all but easy to implement. It has something to do with how patches are applied with git: order matters. 'git log --graph --pretty-oneline' displays s a representation of this order. Git manages pretty well its infamous 'conflicts' when merging branches, applying patches. Not sure it's possible to do that properly with DB updates.
My opinion is that the current system is not that bad. The dependency graph is linear, date ordered by version number last digits. It just becomes crazy when updates are ported from one branch to the other, HEAD to maintenance for example. And there is (am I wrong?) to claim 'communitly' such a number:
http://wiki.koha.org/doku.php?id=en:development:dbrevs:start
It should be used systematically.
Extracting updates code from updatabase.pl script and putting it into a directory containing files named by version number could be a simplification: easier for RM to rename those files if required, not sure. It remains kohastructure.sql...
-- Frédéric _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
My opinion is that the current system is not that bad. The dependency graph is linear, date ordered by version number last digits. It just becomes crazy when updates are ported from one branch to the other, HEAD to maintenance for example. And there is (am I wrong?) to claim 'communitly' such a number:
The problem with the wiki page for db number claiming is it doesn't seem to work, at least it hasn't worked for me in the past. It assumes that the features will be committed in the order that the db numbers are claimed, which I don't believe is true. In that case, if the wiki page were enforced, then db versions would be committed out of order, and the updater would not work correctly. I just don't believe the wiki page is effective at all. After having a number of db version numbers stepped on, I stopped using it.
This is one of the two major concerns I have with the current system, and lead to my proposal. The other concern is proprietary database updates, which are practically impossible with the current system because of this same problem of stomping on revision numbers. On Fri, 2010-01-29 at 13:58 +0100, Paul Poulain wrote:
Frederic Demians a écrit :
Those kind of things can be tricky and messy. Other idea : shouldn't we open a new position, named "database consistency manager" or something like that ?
I don't think having a DB Consistency Manager is going to help that much. I don't think it would be much different from the wiki page. It would be one person in charge instead of several people trying to coordinate. But that person would still have a lot of work to do with changing patches or making patches of their own to keep database updates sane. -- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
My opinion is that the current system is not that bad. The
dependency
graph is linear, date ordered by version number last digits. It just becomes crazy when updates are ported from one branch to the other, HEAD to maintenance for example. And there is (am I wrong?) to claim 'communitly' such a number:
The problem with the wiki page for db number claiming is it doesn't seem to work, at least it hasn't worked for me in the past. It assumes that the features will be committed in the order that the db numbers are claimed, which I don't believe is true. In that case, if the wiki page were enforced, then db versions would be committed out of order, and the updater would not work correctly. I just don't believe the wiki page is effective at all. After having a number of db version numbers stepped on, I stopped using it.
This is one of the two major concerns I have with the current system, and lead to my proposal. The other concern is proprietary database updates, which are practically impossible with the current system because of this same problem of stomping on revision numbers.
On Fri, 2010-01-29 at 13:58 +0100, Paul Poulain wrote:
Frederic Demians a écrit :
Those kind of things can be tricky and messy.
I agree. But they already are. When one works on new features for 3.4 (see RFC for 3.4) and have to
Michael Hafen a écrit : plan for DB updates, how does one deal with update database and rebase and merging ? On the other hand, I agree I proposed things that could become very complicated. But it was brain dumping and proposals. I donot think the wiki has been much helpfull. It is quite difficult to plan how many updates you will need in order to bug fix and design a feature. It could be in fact a simple text file in installer/data/mysql/ ticketsclaim.... But then, simply getting But the fact is that it would not adress the fact that you can run multiple branches on the same codebase. And you would LOVE the merge to be the more seamless as possible. As release maintainer for 3.0 I **had to*** create and arrange db updates in a different order than master branch. Knowing wich db version you were on 3.00.01 and not 3.01 was really important, even though its management could have been improved. (Things that would add some tables and maybe break features should never have mixed up with stable code.) It ended up to be VERY tricky and dangerous to only cherry pick bug fixes unless master code is really close to maintenance branch. This lead to duplication of code. This would have been avoided if updatedatabase30.pl replaced updatedatabase.pl. But then, I would have had to cope with conflict at each cherry pick even if the commit would not update updatedatabase.pl. Not to mention all the db changes that local users may want to do and track.
Other idea : shouldn't we open a new position, named "database consistency manager" or something like that ?
I don't think having a DB Consistency Manager is going to help that much. I don't think it would be much different from the wiki page. It would be one person in charge instead of several people trying to coordinate. But that person would still have a lot of work to do with changing patches or making patches of their own to keep database updates sane.
As release maintainer for 3.0 I **had to*** create and arrange db updates in a different order than master branch. Knowing wich db version you were on 3.00.01 and not 3.01 was really important, even though its management could have been improved. (Things that would add some tables and maybe break features should never have mixed up with stable code.) It ended up to be VERY tricky and dangerous to only cherry pick bug fixes unless master code is really close to maintenance branch. This lead to duplication of code. This would have been avoided if updatedatabase30.pl replaced updatedatabase.pl. But then, I would have had to cope with conflict at each cherry pick even if the commit would not update updatedatabase.pl.
Not to mention all the db changes that local users may want to do and track.
As release manager, having dealt with db updates coordinated with code updates, you're obviously in position to design an functional updates system for Koha. If you could elaborate various scenario describing step-by-step issues you encountered, it may help to design a new system that decrease complexity rather than increase it. -- Frédéric DEMIANS http://www.tamil.fr/u/fdemians.html
There is one great advantage of the current approach, and that is that when a user says "we are on Koha version X", you know exactly what the database state for that should be. You know the tables, fields, constraints AND the codebase. To be clear, that means that you can *do development* specified against that version number. Similarly, the RM only has to know that the next increment applies cleanly to previous one. He does not have to go hunting down edge cases where DB-thread X and public-proprietary-thread Y conflict in the absence of some random other thread Z. I would advise that we should not relinquish this advantage without serious consideration. Once you get into mix/match situation, the version number does NOT tell you the state of the database. The way LibLime handled LEK updates was quite simple. They had a different syspref declaring the internal version number. They batched all the internal updates in a chunk at the end of the main updatedatabase body, using the same kind of logic. You should be able to find a commit by Ryan where he adds some whitespace to updatedatabase to make it even easier to delineate. I suggest if you want to pursue it, that reimplementing that with say LocalCustomVersion would be the simplest approach. When the customizer is ready to commit back to mainline, she can combine all the local DB updates into one current KohaVersion update. For comparison, let me mention DB updates for Evergreen. Only after I started working on it did Evergreen adopt any explicit DB versioning! That has been resolved using a script that processes individual incremental SQL files: http://svn.open-ils.org/trac/ILS/browser/trunk/Open-ILS/src/sql/Pg/upgrade That system is pretty good. However with multiple committers, the "I got next" linearity is still a small problem. So in the end, I think that is just the price you pay for having the version number refer to a known DB state. --Joe Atzberger
Reply inline. On Tue, 2010-02-02 at 16:50 -0500, Joe Atzberger wrote:
There is one great advantage of the current approach, and that is that when a user says "we are on Koha version X", you know exactly what the database state for that should be. You know the tables, fields, constraints AND the codebase. To be clear, that means that you can do development specified against that version number.
In simplest term what I want to do is replace a numeric comparison with a string comparison. The ability to develop again a certain version number is intact, the difference is that the number becomes a string.
Similarly, the RM only has to know that the next increment applies cleanly to previous one. He does not have to go hunting down edge cases where DB-thread X and public-proprietary-thread Y conflict in the absence of some random other thread Z. I would advise that we should not relinquish this advantage without serious consideration. Once you get into mix/match situation, the version number does NOT tell you the state of the database.
This isn't the RM's job, it is the developers. It is currently the developers job to make sure his code is clean with other developers code. That would still be the case either way. As you mention below there are other ways to handle proprietary database versioning. So the problem already exists. I'm sure the solution already exists too, which is for the kind and talented developers to make sure they test edge cases in their code and test new code against proprietary code.
The way LibLime handled LEK updates was quite simple. They had a different syspref declaring the internal version number. They batched all the internal updates in a chunk at the end of the main updatedatabase body, using the same kind of logic. You should be able to find a commit by Ryan where he adds some whitespace to updatedatabase to make it even easier to delineate. I suggest if you want to pursue it, that reimplementing that with say LocalCustomVersion would be the simplest approach. When the customizer is ready to commit back to mainline, she can combine all the local DB updates into one current KohaVersion update.
This is a good suggestion. I like the idea of having a seperate system preference to track proprietary database revisions. It doesn't have the benefit of making public database updates easier to coordinate, because we would still be using a numeric comparison. There would still be version number stomping for the RM to sort out in submitted patches. But I do think it is a good alternative.
For comparison, let me mention DB updates for Evergreen. Only after I started working on it did Evergreen adopt any explicit DB versioning! That has been resolved using a script that processes individual incremental SQL files: http://svn.open-ils.org/trac/ILS/browser/trunk/Open-ILS/src/sql/Pg/upgrade
That system is pretty good. However with multiple committers, the "I got next" linearity is still a small problem. So in the end, I think that is just the price you pay for having the version number refer to a known DB state.
I think we can work around the "I got next" problem by removing the numeric comparison from the equation. It will mean more work for developers in testing their updates. I'll grant that as the price to pay against the "I got next" problem. I'm willing to pay that price.
--Joe Atzberger _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Thanks for your comments. This is the kind of discussion I was hoping for. Here we have a good alternative to my proposal in the separate system preference to track proprietary database revisions. -- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
On Tue, Feb 2, 2010 at 6:30 PM, Michael Hafen <mdhafen@tech.washk12.org>wrote:
On Tue, 2010-02-02 at 16:50 -0500, Joe Atzberger wrote:
There is one great advantage of the current approach, and that is that when a user says "we are on Koha version X", you know exactly what the database state for that should be. You know the tables, fields, constraints AND the codebase. To be clear, that means that you can do development specified against that version number.
In simplest term what I want to do is replace a numeric comparison with a string comparison. The ability to develop again a certain version number is intact, the difference is that the number becomes a string.
It doesn't matter whether it is a string or not. Once you remove the linearity of the update process, you no longer know what revision X means. You could have got there by applying X to: - 3.00.04.019, - 3.02.01.555, - "Z", or even - "2.00.00.01". It means you have applied patch X, it does not mean *database state* X. The linearity of the process is why we happen to use numbers to describe it, since numbers have an agreed-upon order. That's why your approach would need a log of every update applied AND the order of application instead of just the version label. Because if one revision "A" creates a table and another "B" drops it if it exists, you get totally different results if you run them AB vs. BA. Currently 3.00.04.019 implies 3.00.04.018, 3.00.04.017, etc. The proposed scheme forces us to know the entire contents of the upgrade log to know the database state. That is to say, you need to READ the database, to know what the database should look like. This violates some distinctions we would otherwise like to retain between structure and data.
Similarly, the RM only has to know that the next increment applies
cleanly to previous one. He does not have to go hunting down edge cases where DB-thread X and public-proprietary-thread Y conflict in the absence of some random other thread Z. I would advise that we should not relinquish this advantage without serious consideration. Once you get into mix/match situation, the version number does NOT tell you the state of the database.
This isn't the RM's job, it is the developers. It is currently the developers job to make sure his code is clean with other developers code. That would still be the case either way.
As you mention below there are other ways to handle proprietary database versioning. So the problem already exists. I'm sure the solution already exists too, which is for the kind and talented developers to make sure they test edge cases in their code and test new code against proprietary code.
It cannot be the developers job to know all other developers' unreleased changes. And it IS the RM's job to make sure each DB revision applies cleanly to the previous one. In the proposed scheme, instead of "to the previous one" that becomes "to any possible previous state". And even worse, "to any hypothetical state yet-to-be developed"! There is no guarantee that somebody else won't have made 100 revisions before trying to apply the current one. It will be a much harder job, whoever is doing it, combining complexity and obscurity. The result will be that the tester tests the previous state they happen to have because it is ridiculous to imagine that they must then go test against every DB schema being developed, or ever used, when all we should care about is getting it to apply to one target (at HEAD). I think we can work around the "I got next" problem by removing the
numeric comparison from the equation.
Numeric or not, removing the linearity of updates means that "next" does not exist. The problem is that "previous" stops existing also. I'm not against considering new approaches to DB versioning, but on it's own this is not a suitable alternative to me. I realize I haven't been pushing code towards Koha for maybe half a year now, but I honestly don't think that DB revisions were a big enough problem to have us abandon the advantages of the current approach without a more pronounced incentive. --Joe
Hi, 2010/2/3 Joe Atzberger <ohiocore@gmail.com>:
On Tue, Feb 2, 2010 at 6:30 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote: I realize I haven't been pushing code towards Koha for maybe half a year now, but I honestly don't think that DB revisions were a big enough problem to have us abandon the advantages of the current approach without a more pronounced incentive.
Quickly, from my point of view the main things I'd like to see in a revised database update system are: * the ability to end up with a linear set of database updates in released versions, for the reasons Joe mentions * secondarily, the ability to have a linear path in HEAD * a reduction in purely textual merge conflicts - splitting up the monolithic updatedatabase.pl would go a long way to address this * a better way to manage database updates that are backported to the maintenance branch * a mechanism to allow a developer to readily renumber updates that they're working on in a topic branch when it comes time to submit to HEAD * getting simple system preference updates out of the main Regards, Galen -- Galen Charlton gmcharlt@gmail.com
I have an interesting idea about the system preference request in the last point here. Have the system preference editor create in the database a row for any pref's it didn't find there. It would have to keep track of what the default value should be for all pref's though. With the new system preference editor, as I understand it, changing the text of a pref is changing a file instead of the database now, so that's one of the two cases I can think of for a system preference to change the database. The other case is to create or delete a system preference. Let's put aside creating for a moment. Thinking about deleting a pref, I might argue that it shouldn't lightly be taken out of the database. Currently is is removed from the system preference editor, and ends up in the Local Use category. Maybe this category could be renamed 'Local Use / Deprecated'. The problem then becomes that the system preference editor isn't run often enough. So maybe there needs to be an addition to the version check in Auth.pm. Along with checking kohaversion.pl it could also check something like 'sysprefs.pl' to make sure all officially supported system preferences are in the database. Then the update mechanism could create the system preference instead while asking for it to be set ( if the user doesn't want to use the default setting ). That might be the way to go. Have a 'sysprefs.pl' that checks the database, and add to the update mechanism to create any that it reports as not being there. That way developers would be changing a file(s) instead of the database. On Wed, 2010-02-03 at 11:59 -0500, Galen Charlton wrote:
Hi,
Quickly, from my point of view the main things I'd like to see in a revised database update system are:
* the ability to end up with a linear set of database updates in released versions, for the reasons Joe mentions * secondarily, the ability to have a linear path in HEAD * a reduction in purely textual merge conflicts - splitting up the monolithic updatedatabase.pl would go a long way to address this * a better way to manage database updates that are backported to the maintenance branch * a mechanism to allow a developer to readily renumber updates that they're working on in a topic branch when it comes time to submit to HEAD * getting simple system preference updates out of the main
Regards,
Galen
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
I'm still thinking about this. The only thing I've come up with so far is to enclose all the database updates in a subroutine, and put the code from kohaversion.pl into updatedatabase.pl. That would make one less file to edit. And if the few sub's at the end of updatedatabase.pl were moved to the top of the file, and the kohaversion stuff put at the bottom, then the sql would end up right above the version number. That way it might be more likely that the version number would be kept up to date with the sql. The other thought I have is to dump the updatedatabase.pl entirely. Have a file that lists what the system preferences are supposed to look, and another file that lists what the database is supposed to look like. Then when the version number changes these files can be run to check if the actual sys.pref. / database state matches the expected sys.pref. / database state. This would be much more complex though, especially the database stuff. If I know off hand how to easily find out exactly how a certain column looks, like it's type and such, this would be easier. We'd just have to make the script able to compose sql to add / drop / modify columns. It's very complex, but makes a nice option. There would be no question about the database state for a given version number. On the other hand this would make database changes very difficult to backport to the maintenance branches. So that's where my thoughts are at the moment. Any comments here? On Wed, 2010-02-03 at 11:59 -0500, Galen Charlton wrote:
Hi,
2010/2/3 Joe Atzberger <ohiocore@gmail.com>:
On Tue, Feb 2, 2010 at 6:30 PM, Michael Hafen <mdhafen@tech.washk12.org> wrote: I realize I haven't been pushing code towards Koha for maybe half a year now, but I honestly don't think that DB revisions were a big enough problem to have us abandon the advantages of the current approach without a more pronounced incentive.
Quickly, from my point of view the main things I'd like to see in a revised database update system are:
* the ability to end up with a linear set of database updates in released versions, for the reasons Joe mentions * secondarily, the ability to have a linear path in HEAD * a reduction in purely textual merge conflicts - splitting up the monolithic updatedatabase.pl would go a long way to address this * a better way to manage database updates that are backported to the maintenance branch * a mechanism to allow a developer to readily renumber updates that they're working on in a topic branch when it comes time to submit to HEAD * getting simple system preference updates out of the main
Regards,
Galen
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
I've been assuming all this time, without stating the assumption, that new database changes would only be appended to the list in updatedatabase.pl. I thought that would guarantee the linearity that you are talking about. Of course it is important to have this linearity fro the reasons you have mentioned. Maybe my assumption is unreasonable since it doesn't enforce linearity. With Galen's requests to take into account to this become a very difficult problem to solve. We need a system that enforces itself, but is flexible. Gonna have to think about that alot. I think the key is is Galen's fifth point: being able to easily renumber updates before submission to HEAD. On Wed, 2010-02-03 at 11:43 -0500, Joe Atzberger wrote:
It doesn't matter whether it is a string or not. Once you remove the linearity of the update process, you no longer know what revision X means. You could have got there by applying X to: * 3.00.04.019, * 3.02.01.555, * "Z", or even * "2.00.00.01". It means you have applied patch X, it does not mean database state X. The linearity of the process is why we happen to use numbers to describe it, since numbers have an agreed-upon order. That's why your approach would need a log of every update applied AND the order of application instead of just the version label. Because if one revision "A" creates a table and another "B" drops it if it exists, you get totally different results if you run them AB vs. BA.
Currently 3.00.04.019 implies 3.00.04.018, 3.00.04.017, etc. The proposed scheme forces us to know the entire contents of the upgrade log to know the database state. That is to say, you need to READ the database, to know what the database should look like. This violates some distinctions we would otherwise like to retain between structure and data. --Joe
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
As release maintainer for 3.0 I **had to*** create and arrange db updates in a different order than master branch. Knowing wich db version you were on 3.00.01 and not 3.01 was really important, even though its management could have been improved. (Things that would add some tables and maybe break features should never have mixed up with stable code.) It ended up to be VERY tricky and dangerous to only cherry pick bug fixes unless master code is really close to maintenance branch. This lead to duplication of code. This would have been avoided if updatedatabase30.pl replaced updatedatabase.pl. But then, I would have had to cope with conflict at each cherry pick even if the commit would not update updatedatabase.pl.
Not to mention all the db changes that local users may want to do and track.
As release manager, having dealt with db updates coordinated with code updates, you're obviously in position to design an functional updates system for Koha. If you could elaborate various scenario describing step-by-step issues you encountered, it may help to design a new system that decrease complexity rather than increase it.
Frederic Demians a écrit :
Those kind of things can be tricky and messy. Other idea : shouldn't we open a new position, named "database consistency manager" or something like that ?
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Frederic Demians <frederic@tamil.fr>
Those kind of things can be tricky and messy. Something like a dependency graph would be required, which is all but easy to implement. It has something to do with how patches are applied with git: [...]
I think a different way of handling database updates is a good idea because it would remove the current serious chilling effect of claiming a dbrev when we're not sure if we have booked enough time to finish tidying, testing and sending (there's often some subtle bug shows up on the final peer review, isn't there?). Most systems running koha will have at least two depedency graph solvers installed: make and git. If we could harness one of those, that would be brilliant. Hope that helps, -- MJ Ray (slef) Webmaster and LMS developer at | software www.software.coop http://mjr.towers.org.uk | .... co IMO only: see http://mjr.towers.org.uk/email.html | .... op
participants (11)
-
Chris Nighswonger -
Frederic Demians -
Frédéric DEMIANS -
Galen Charlton -
Joe Atzberger -
Kyle Hall -
LAURENT Henri-Damien -
LAURENT Henri-Damien -
Michael Hafen -
MJ Ray -
Paul Poulain