Jenkins status [was Jenkins build is back to stable : Koha_master #1475]
Hi all! I often receive emails from Jenkins: "master is unstable", "master is back to stable". Last unstable status was caused by a fail in t/db_dependent/Acquisition/Invoices.t (I think). Here Jenkins says : "it is stable, see changes". But changes are about a tt change. I don't know who fixed the failure and how it was fixed. It is frustrating not to know :) Especially because the tests pass on master with my data! Do you think it would be possible to put something in place in order to follow Jenkins ' moods? For discussion. Regards, Jonathan 2013/10/23 <chris@bigballofwax.co.nz>:
See <http://jenkins.koha-community.org/job/Koha_master/1475/changes>
No answer? Does that mean I am the only one wondering why jenkins is back to stable? Or maybe is it because I was not clear enough in my question ? A big work has been made on unit tests. It seems important to me to know why our integration server fails when the code passes on my development instance. Regards, Jonathan 2013/10/24 Jonathan Druart <jonathan.druart@biblibre.com>:
Hi all!
I often receive emails from Jenkins: "master is unstable", "master is back to stable". Last unstable status was caused by a fail in t/db_dependent/Acquisition/Invoices.t (I think). Here Jenkins says : "it is stable, see changes". But changes are about a tt change. I don't know who fixed the failure and how it was fixed. It is frustrating not to know :) Especially because the tests pass on master with my data!
Do you think it would be possible to put something in place in order to follow Jenkins ' moods?
For discussion. Regards, Jonathan
2013/10/23 <chris@bigballofwax.co.nz>:
See <http://jenkins.koha-community.org/job/Koha_master/1475/changes>
Your question is certainly valid. There are still too many tests depending on some data to exist. Finding out why it no longer fails requires some (extensive?) digging into which patches were pushed in the meantime. ________________________________________ Van: koha-devel-bounces@lists.koha-community.org [koha-devel-bounces@lists.koha-community.org] namens Jonathan Druart [jonathan.druart@biblibre.com] Verzonden: woensdag 30 oktober 2013 16:59 Aan: koha-devel@lists.koha-community.org Onderwerp: Re: [Koha-devel] Jenkins status [was Jenkins build is back to stable : Koha_master #1475] No answer? Does that mean I am the only one wondering why jenkins is back to stable? Or maybe is it because I was not clear enough in my question ? A big work has been made on unit tests. It seems important to me to know why our integration server fails when the code passes on my development instance. Regards, Jonathan 2013/10/24 Jonathan Druart <jonathan.druart@biblibre.com>:
Hi all!
I often receive emails from Jenkins: "master is unstable", "master is back to stable". Last unstable status was caused by a fail in t/db_dependent/Acquisition/Invoices.t (I think). Here Jenkins says : "it is stable, see changes". But changes are about a tt change. I don't know who fixed the failure and how it was fixed. It is frustrating not to know :) Especially because the tests pass on master with my data!
Do you think it would be possible to put something in place in order to follow Jenkins ' moods?
For discussion. Regards, Jonathan
2013/10/23 <chris@bigballofwax.co.nz>:
See <http://jenkins.koha-community.org/job/Koha_master/1475/changes>
Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Your question is certainly valid. There are still too many tests depending on some data to exist. Finding out why it no longer fails requires some (extensive?) digging into which patches were pushed in the meantime.
Jenkins is usually back to stable without any patch pushed. So an intervention is done on the Jenkins DB on adding/updating/removing data.
* Jonathan Druart (jonathan.druart@biblibre.com) wrote:
Your question is certainly valid. There are still too many tests depending on some data to exist. Finding out why it no longer fails requires some (extensive?) digging into which patches were pushed in the meantime.
Jenkins is usually back to stable without any patch pushed. So an intervention is done on the Jenkins DB on adding/updating/removing data.
Not usually no, often the test is failing due to depending on external data sources. XISBN.t is the prime example, it can fail because it can't connect to the external data source. The current failing test is Serials http://jenkins.koha-community.org/job/Koha_master/lastCompletedBuild/testRep... 01:10:42.908 # Failed test 'test getting history from sub-scription' 01:10:42.909 # at t/db_dependent/Serials.t line 59. 01:10:42.915 01:10:42.915 # Failed test 'Subscription has at least one serial' 01:10:42.916 # at t/db_dependent/Serials.t line 62. This one will be to do with data, and changes it started failing http://jenkins.koha-community.org/job/Koha_master/1482/ When all the serials changes were pushed. It fails on my local machine too Test Summary Report ------------------- t/db_dependent/Serials.t (Wstat: 768 Tests: 35 Failed: 3) Failed tests: 6, 8-9 Non-zero exit status: 3 Files=1, Tests=35, 0 wallclock secs ( 0.04 usr 0.00 sys + 0.43 cusr 0.02 csys = 0.49 CPU) Result: FAIL Chris -- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand
------------------- t/db_dependent/Serials.t (Wstat: 768 Tests: 35 Failed: 3) Failed tests: 6, 8-9 Non-zero exit status: 3 Files=1, Tests=35, 0 wallclock secs ( 0.04 usr 0.00 sys + 0.43 cusr 0.02 csys = 0.49 CPU) Result: FAIL
This was fixed by http://jenkins.koha-community.org/job/Koha_master/1487/changes Chris
Hello, Thank you for your answer Chris ! So there are 3 kinds of failure causes: 1/ C4::XISBN::get_xisbns fails because the connection to an external data source cannot be established. 2/ data is not created by the unit tests and does not exist in the sample files. 3/ a test "normally" fails For 1, is it consistent to mock the connection ? For 2, I think it could be resolved using the idea from bug 10337: reset the DB data before launching tests. It would be great to receive mails from Jenkins saying the master is unstable only if is a code issue (and not a connection issue or a lack of data). Moreover, we have some unit test files which are never launched. So if a regression is introduced on routines covered by these unit tests, we are not alerted. The prove command should be launched with the recursive flag. Regards, Jonathan 2013/10/30 Chris Cormack <chrisc@catalyst.net.nz>:
* Jonathan Druart (jonathan.druart@biblibre.com) wrote:
Your question is certainly valid. There are still too many tests depending on some data to exist. Finding out why it no longer fails requires some (extensive?) digging into which patches were pushed in the meantime.
Jenkins is usually back to stable without any patch pushed. So an intervention is done on the Jenkins DB on adding/updating/removing data.
Not usually no, often the test is failing due to depending on external data sources. XISBN.t is the prime example, it can fail because it can't connect to the external data source.
The current failing test is Serials http://jenkins.koha-community.org/job/Koha_master/lastCompletedBuild/testRep...
01:10:42.908 # Failed test 'test getting history from sub-scription' 01:10:42.909 # at t/db_dependent/Serials.t line 59. 01:10:42.915 01:10:42.915 # Failed test 'Subscription has at least one serial' 01:10:42.916 # at t/db_dependent/Serials.t line 62.
This one will be to do with data, and changes it started failing http://jenkins.koha-community.org/job/Koha_master/1482/
When all the serials changes were pushed.
It fails on my local machine too Test Summary Report ------------------- t/db_dependent/Serials.t (Wstat: 768 Tests: 35 Failed: 3) Failed tests: 6, 8-9 Non-zero exit status: 3 Files=1, Tests=35, 0 wallclock secs ( 0.04 usr 0.00 sys + 0.43 cusr 0.02 csys = 0.49 CPU) Result: FAIL
Chris -- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand
Hi, On Thu, Oct 31, 2013 at 8:29 AM, Jonathan Druart < jonathan.druart@biblibre.com> wrote:
So there are 3 kinds of failure causes: 1/ C4::XISBN::get_xisbns fails because the connection to an external data source cannot be established. 2/ data is not created by the unit tests and does not exist in the sample files. 3/ a test "normally" fails
For 1, is it consistent to mock the connection ?
Mocking the connection is an option, but IMO it's better to attempt a real connection, and have the test skip if the connection cannot be established.
For 2, I think it could be resolved using the idea from bug 10337: reset the DB data before launching tests.
I think some tweaking is needed for that proposal. It's one thing to have Jenkins reset its database(s) before running a build, but I feel strongly that we should encourage ordinary developers to feel free to run prove -v t/db_dependent/whatever.t at any time against their main development database. We're pretty close to that being safe now, and once all of the DB-dependent tests run in a transaction, it will be safe enough for a developer to run any and all of the tests at will. That's not to say that there shouldn't be scripts available for a developer to set up a separate testing database and easily reset it at will, too, but I don't think they belong as a mandatory test under t/db_dependent. It would be great to receive mails from Jenkins saying the master is
unstable only if is a code issue (and not a connection issue or a lack of data).
Well, there is always going to be the possibility of a problem with the test system, so it's never going to be perfect. For example, if nothing else there will always be the possibility that the I think it might be better if we give folks the option to subscribe to emails from Jenkins, not get them automatically if one of their patches just happens to be in a set that triggered a test failure. RMs and RMaints need to care about Jenkins test results. QA team members should also care. So should ordinary developers, but that doesn't necessarily mean that they have to unconditionally get the nags from Jenkins.
Moreover, we have some unit test files which are never launched. So if a regression is introduced on routines covered by these unit tests, we are not alerted. The prove command should be launched with the recursive flag.
That would be a goal, but because of dependencies, particularly the Solr stuff, we're not yet at the point where prove -r t/, when run on a Jenkins build server, will not result in test failures. If there are specific tests that you've noticed that aren't being run, please enumerate them and I can adjust the Jenkins config. Regards, Galen -- Galen Charlton Manager of Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
Galen, First, thank you for this complete answer.
I think some tweaking is needed for that proposal. It's one thing to have Jenkins reset its database(s) before running a build, but I feel strongly that we should encourage ordinary developers to feel free to run prove -v t/db_dependent/whatever.t at any time against their main development database. We're pretty close to that being safe now, and once all of the DB-dependent tests run in a transaction, it will be safe enough for a developer to run any and all of the tests at will.
I don't think we are closed to offer a testing suite available for all data. We are strongly dependent on data. Examples: library, framework, etc. For the moment the unit tests don't work at all with an UNIMARC DB. For new developers and testers, it is important to know how launching unit tests. With this script, we will tell them: "launch build_a_db_for_ut.pl and launch a prove on any .t file and tests should pass!"
That's not to say that there shouldn't be scripts available for a developer to set up a separate testing database and easily reset it at will, too, but I don't think they belong as a mandatory test under t/db_dependent.
Yes, you are right. In misc/devel?
If there are specific tests that you've noticed that aren't being run, please enumerate them and I can adjust the Jenkins config.
At least all files in a directory in t/db_dependent: t/db_dependent/Acquisition/* t/db_dependent/Circulation/* t/db_dependent/Items/* t/db_dependent/Members/* t/db_dependent/Reports/* t/db_dependent/Serials/* Regards, Jonathan
Hi, On Mon, Nov 4, 2013 at 10:12 AM, Jonathan Druart < jonathan.druart@biblibre.com> wrote:
I don't think we are closed to offer a testing suite available for all data. We are strongly dependent on data. Examples: library, framework, etc. For the moment the unit tests don't work at all with an UNIMARC DB.
Actually, I think this is a bit too strong of a statement, although I intend to set up a fr-UNIMARC database soonish to see how much breakage there is. Obviously, there are a number of tests that implicitly or explicitly assume MARC21 is in effect, and those need to be generalized, but many of the tests need only create some stub bibs and items -- and we could write a routine that replaces the various add hoc ways of doing that with a central way of getting back a stub bib record in either MARC21 or UNIMARC (or NORMARC).
At least all files in a directory in t/db_dependent: t/db_dependent/Acquisition/* t/db_dependent/Circulation/* t/db_dependent/Items/* t/db_dependent/Members/* t/db_dependent/Reports/* t/db_dependent/Serials/*
I've configured the Koha_master and master_maria Jenkins projects to include these tests, and at the moment, they're all passing. Regards, Galen -- Galen Charlton Manager of Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
participants (5)
-
Chris Cormack -
Chris Cormack -
Galen Charlton -
Jonathan Druart -
Marcel de Rooy