On 12/09/2007, at 11:51 AM, MJ Ray wrote:

Rick Welykochy <rick@praxis.com.au> wrote: [...]

...

To stop this problem once and for all from ever occurring again, catch all errors. To catch all errors and thus produce the correct headers for EVERY Koha script, use exception handling. Works a treat :)

What we should do is make the errors prettier. We should probably use CGI::Carp qw(fatalsToBrowser); - is that OK by everyone and do we need this only in C4::Output or in all cgi-bin scripts?

Ok by me, and I think it might need to be in each script, but i'm not sure. Ill try it out and see :) Mason has just suggested to me that we could do it with a syspref. So you could turn it off if you wanted. (I know there is some school of thought that exposing errors in the browser is a security risk).

I don't think exceptions are the way to go. Read the top of perl's exceptions library: [...] # This library is no longer being maintained, and is included for backward # compatibility with Perl 4 programs which may require it. # # In particular, this should not be used as an example of modern Perl # programming techniques. [...]

Also, we're never going to catch all errors. Sysadmins and users sometimes do surprising things, like transposing fields in the configuration, or installing new firewalls.

Yep, very good points Chris -- Chris Cormack chris.cormack@liblime.com VP Research and Development www.liblime.com LibLime +64 21 542 131

Rick Welykochy <rick@praxis.com.au> wrote:

MJ Ray wrote:

...

What we should do is make the errors prettier. We should probably use CGI::Carp qw(fatalsToBrowser); - is that OK by everyone and do we need this only in C4::Output or in all cgi-bin scripts?

This is okay, but it does present a security concern, since internal details of the Perl environment and MySQL could be revealed with the dumped errors.

What details dumped could be exploited? Chris mentioned testing this - what would a Koha error actually dump? I tried it with a silly example and it looks like we just get this by default: Software error: aiee at /home/mjr/webapps/sites/koha/intranet/cgi-bin/mainpage.pl line 17. For help, please send mail to the webmaster (info@ttllp.co.uk), giving this error message and the time and date of the error. So as long as no-one is putting passwords into their die messages (which would be a bug already, because the password would appear in logs), then I think it looks like we should be OK. Also, note that we can configure and/or subclass CGI::Carp to pretty up the message and suppress sensitive info.

In my experience, system and internal errors are for "programmers eyes only".

In my experience, Koha sysadmins aren't finding the system and internal errors, so they never reach the programmers' eyes. How many times do we post some variation of "check koha-error_log" to the koha lists? How many people don't do that? How many bugs are going unfixed because we're not making it easy for people to send us good bug reports the first time?

Would you want these errors to be displayed on the browser of an OPAC user?

No. They should be disabled when putting the system into production, or maybe OPAC should be configured differently. I think I want these errors to be displayed on the browser of a Librarian Intranet user, and displayed on the OPAC during setup and testing. This would get us straight past the "what is the problem?" stage and onto "why has this happened?" [...]

Fatal errors are raised as exception via die and can be caught via eval. Exceptions (i.e. fatal error processing) are indeed very modern Perl programming techniques.

So by exceptions, you did not mean exceptions as such. Thanks for clarifying. The manual entry for die suggests using Carp, which is what I suggested too. Using core perl modules seems far safer than relying on everyone remembering on wrapping everything that can go wrong in an appropriate eval{}. Sod's law says that an error would occur outside the eval{} anyway. I'm slightly scared that you have re-invented that wheel and then used that re-invention in large-scale ecommerce systems. Why aren't you using the Carp modules or something based on them? Puzzled, -- MJ Ray - see/vidu http://mjr.towers.org.uk/email.html Experienced webmaster-developers for hire http://www.ttllp.co.uk/ Also: statistician, sysadmin, online shop builder, workers co-op. Writing on koha, debian, sat TV, Kewstoke http://mjr.towers.org.uk/

On 9/12/07, Rick Welykochy <rick@praxis.com.au> wrote:

...

The manual entry for die suggests using Carp, which is what I suggested too. Using core perl modules seems far safer ...

The "perldoc -f die" entry does not mention Carp.

Carp provides better formatting of message sent via die:

"The Carp routines are useful in your own modules because they act like die() or warn(), but with a message which is more likely to be useful to a user of your module."

i.e. Carp is for terminating (via croak or confess) or warning (via carp or cluck) in a "nicer way", to be used for the benefit of CLIENTS of the software you are writing.

You are looking at Carp instead of CGI::Carp. The latter is adapted to the task at hand (OPAC errors).

Why aren't you using the Carp modules or something based on them?

Carp does not provide exception catching.

But CGI::Carp can, and commonly does. The following catches fatal errors, including "die": use CGI::Carp qw(fatalsToBrowser); The following catches warnings, including "warn": use CGI::Carp qw(warningsToBrowser); The messages get formatted and datestamped as appropriate for the logs and also (optionally) displayed to the browser. I find it very useful, and it avoids the performance hit of having to defer compilation of everything to runtime evals. Carp does not provide the functionality I have demonstrated at

http://praxis.com.au/demo_error.pl/showio

which intercepts an IO error, and displays detailed information to the browser in addition to notifying the sysadmin.

The demo_error.pl example intercepts ALL errors and works for all perl applications :)

I suspect upon examination you will agree such functionality is indeed present in CGI::Carp, with the same broad applicability. Another difference is that CGI::Carp would allow you to, for example, toggle the browser notification at runtime, whereas the evals have to be built into the code. Leaving aside the question of functionality as largely settled, I want to revisit the question of security briefly. I would suggest to MJ Ray and the rest of the list that fatalsToBrowser error messages are certainly helpful... helpful enough to constitute an increased security exposure. Dependent on the implementation of course, a skilled observer could be able to discern: - otherwise encapsulated file structure outside DOCROOT, - presence and location of system executables, - anything in $!, including STDERR from system calls, - variable names and types, their relationship to query params, and the type of taint checking performed on them... or LACK of taint checking, etc. Then it isn't much of a leap to start trying to overload a POST with something like:

(??{$later_evaled_var="rm -f /file"})

That being said, I don't think the security issues are overriding. The main problems would be where fatalsToBrowser was just dropped on top of system code. Properly designed CGI would mitigate most of the additional exposure. I find using CGI:Carp accelerates the turnaround on bugfixing my own code, from both the log formatting and the delivery of a more sensible error message to the user (that also happens to match the logs). In not very long, I hope to be able to contribute more directly to the OPAC codebase and help address this kind of problem. --Joe Atzberger INFOhio Technical Specialist Metropolitan Educational Council

MJ Ray wrote:

And what does that script do when it intercepts it? It displays a better-formatted system error, dumping lots of internal data! Using a Carp is more general and would reveal less about the internals by default.

I think you are taking my published example too literally: it just shows what can be done with exception processing. And dumping that stuff is very handy during software development. Why do you think Java does similar? It makes debugging much saner. In a production environment, dumping all that stuff to the browser is disabled. Right?

Also, what happens if your script contains an error outside the eval{}? Sod's law says that some errors would occur outside the eval{} anyway. Putting a use Carp at the top handles the whole script. Only a perl core problem would derail that and that's beyond our powers.

This is true. Any exceptions raised outside the eval are of course not caught. I don't know how good CGI::Carp is at catching "compile-time" or startup errors. Some experimentation may be required. But look at it this way. During development anything can and often does happen. The production environment is a different beastie. If your application cannot make it to the eval { ... } that wraps a script, the application and its attendant services are going down bigtime.

If we can recover gracefully from a resource failure, we should test for it and handle it in the code, as usual, not get into the half-hidden couplings of throwing and catching exceptions. (Look up about exceptions and coupling in programming/software engineering texts and journals if you've not noticed this problem. I expect someone has posted something with a title like "Exceptions considered harmful" on the web by now.)

But this is exactly how CGI::Carp works. By catching exceptions. And it is probably a reasonable and simple solution for Koha. But it will be hard to turn CGI:Carp on in development and off in production. Remember: most of the errors we see on the Koha list from library users are during installation and deployment, not during actual production. Detecting and listing errors is not and should hopefully not be required in production, unless the code is flawed. It is glib to say "Exceptions considered evil". Yes, I agree that code sprinkled willy-nilly with exception handling blocks can become tedious and a maintenance nightmare. That is not what is being discussed here. You've "raised" a red herring, so to speak. Horses for courses: CGI::Carp as well as my published solution use ONE single eval to catch all errors that may come from the application. That's all. Just one. Regarding error codes vs. exceptions. Each has its place. I prefer error codes for simple, recoverable situations and exceptions for fatal errors. When an error is fatal, i.e. like an assertion failure or hardware fault, there is absolutely no point in continuing execution of the code: program abort is about all that can be done, since the state of things after the fatal error is undefined. Let's not lose sight of why this thread was opened. Myself and many many other Koha installers have been stymied and mystified when an error occurs and all they see in the error_log is "Unexpected end of script headers", or we are offered the script as a download. Sometimes we have had to run the script in the shell to see what the error message was. This can be easily solved by using CGI::Carp or some other exception catch, which ensures that the correct and complete headers are written to the client. I've used CGI::Carp with great success. I roll my own exception handler when I need to auto-log the error and perhaps email a sys admin that an unexpected problem has occurred. The Koha developers can decide how much intervention and reportage is required when a fatal error occurs. CGI::Carp would be a vast improvement over nothing. cheers rickw -- _________________________________ Rick Welykochy || Praxis Services The idea that Bill Gates has appeared like a knight in shining armour to lead all customers out of a mire of technological chaos neatly ignores the fact that it was he who, by peddling second-rate technology, led them into it in the first place. -- Douglas Adams on Windows '95.

Joe Atzberger wrote:

On 9/13/07, *Rick Welykochy* <rick@praxis.com.au <mailto:rick@praxis.com.au>> wrote:

But it will be hard to turn CGI:Carp on in development and off in production.

Actually, it seems pretty easy to me.

use CGI::Carp qw(fatalsToBrowser); ... # some code ... fatalsToBrowser(0); # off ... # more code ... fatalsToBrowser(1); # back on

So you can decide to turn it on or off at runtime conditionally, based on whatever you want. You wouldn't have to move code around at all.

Now we are getting somewhere. But we don't want a solution where the library installer has to edit perl source code, do we? Suppose my library's security policy is to disallow any errors or dumps of same to the browser. In this production case, ALL fatals to browser in all Koha scripts must be disabled. Rather than forcing the Koha sys admin / installer to edit all the scripts, this is my proposal: in a typical koha_whatever.pl script we have: use CGI::Carp qw(fatalsToBrowser); # trap errors to error_log and the browser : : fatalsToBrowser(0) if !Koha::Config->fatalstobrowser; # trap errors only to error_log if so configured This enables any Koha implementation to (a) see all fatal errors in error_log at all times and (b) disable the viewing of same in the browser if they see fit. All that is then needed is a new entry in /etc/koha.conf fatalstobrowser=1 and all the installer needs to do once testing is complete and the system is ready for production (if they want): fatalstobrowser=0 cheers rickw -- _________________________________ Rick Welykochy || Praxis Services Traditionally, most of Australia's imports come from overseas. -- Keppel Enderbery