[Koha-devel] Jenkins status [was Jenkins build is back to stable : Koha_master #1475]

[Galen Charlton]

Hi,

On Thu, Oct 31, 2013 at 8:29 AM, Jonathan Druart <
jonathan.druart at biblibre.com> wrote:

> So there are 3 kinds of failure causes:
> 1/ C4::XISBN::get_xisbns fails because the connection to an external
> data source cannot be established.
> 2/ data is not created by the unit tests and does not exist in the sample
> files.
> 3/ a test "normally" fails
>
> For 1, is it consistent to mock the connection ?
>

Mocking the connection is an option, but IMO it's better to attempt a real
connection, and have the test skip if the connection cannot be established.


> For 2, I think it could be resolved using the idea from bug 10337:
> reset the DB data before launching tests.
>

I think some tweaking is needed for that proposal.  It's one thing to have
Jenkins reset its database(s) before running a build, but I feel strongly
that we should encourage ordinary developers to feel free to run prove -v
t/db_dependent/whatever.t at any time against their main development
database.  We're pretty close to that being safe now, and once all of the
DB-dependent tests run in a transaction, it will be safe enough for a
developer to run any and all of the tests at will.

That's not to say that there shouldn't be scripts available for a developer
to set up a separate testing database and easily reset it at will, too, but
I don't think they belong as a mandatory test under t/db_dependent.

It would be great to receive mails from Jenkins saying the master is
> unstable only if is a code issue (and not a connection issue or a lack
> of data).
>

Well, there is always going to be the possibility of a problem with the
test system, so it's never going to be perfect.  For example, if nothing
else there will always be the possibility that the

I think it might be better if we give folks the option to subscribe to
emails from Jenkins, not get them automatically if one of their patches
just happens to be in a set that triggered a test failure.

RMs and RMaints need to care about Jenkins test results.  QA team members
should also care.  So should ordinary developers, but that doesn't
necessarily mean that they have to unconditionally get the nags from
Jenkins.


> Moreover, we have some unit test files which are never launched. So if
> a regression is introduced on routines covered by these unit tests, we
> are not alerted.
> The prove command should be launched with the recursive flag.
>

That would be a goal, but because of dependencies, particularly the Solr
stuff, we're not yet at the point where prove -r t/, when run on a Jenkins
build server, will not result in test failures.

If there are specific tests that you've noticed that aren't being run,
please enumerate them and I can adjust the Jenkins config.

Regards,

Galen
-- 
Galen Charlton
Manager of Implementation
Equinox Software, Inc. / The Open Source Experts
email:  gmc at esilibrary.com
direct: +1 770-709-5581
cell:   +1 404-984-4366
skype:  gmcharlt
web:    http://www.esilibrary.com/
Supporting Koha and Evergreen: http://koha-community.org &
http://evergreen-ils.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20131031/221a1133/attachment.html>