[Koha-devel] RFC: setting up a test server

[Andrew Moore]

Hi Koha Developers -

I'm interested in setting up a system to run our test suite regularly
and alert us of the results. You may know this as a smoke tester or a
continuous integration server, but essentially, it's a thing that runs
the test suite regularly and lets us know when things pass and fail.
Here's how I picture it working, and please let me know if you have
any suggestions or questions about it.

I have set up a smolder server at
<http://arwen.metavore.com:8000/app/public_projects/smoke_reports/1>
that will accept test reports and display them with pretty red,
yellow, and green boxes and other information. I expect my nightly
test results to go in there, and if you or others would like to send
test results there, that would be great. That's where we can look to
make sure that stuff we added yesterday didn't break other features in
other places in the code.

To get the reports there, I just submitted a patch
<http://markmail.org/message/upa6j34f6rwegsgr> for review that adds a
few targets to the t/Makefile and a perl program to the t/ directory.
These are so that you can run "make submit" from the t/ directory in
the distribution to send your test results to the smolder server. I'll
set up a cronjob to run the test suite on the latest git repository
each night and submit the results to the smolder server.

Based on any discussion that this email opens up, I may modify that
patch and submit it intending for it to get applied. I've opened up
bug 2498 <http://bugs.koha.org/cgi-bin/bugzilla/show_bug.cgi?id=2498>
to discuss this. There's also a page on the wiki that I started to
describe these tools and how they can be used.
<http://wiki.koha.org/doku.php?id=en:development:smoke_reports>

You'll notice that none of this actually requires you or any other
developers to change their behavior in any way. If you don't like this
or have some kind of moral or religious objection to it, you can just
ignore all of this smolder and smoke stuff and do just fine.

So, let me know what you think, and expect the smoke test results to
start showing up in smolder soon.

Thanks,
-Andy