One of the things I am researching a bit these days is how best to go about incorporating logging into a web based application.  I am looking at it from both an application troubleshooting perspective as well as a security perspective. 

Obviously web servers log activity in the W3C format. But those are access logs which tend to get very detailed and large. And it really does not give you a good picture of what is going on in the application. I've been doing a fair bit of reading in preparation for this and have come across some good information.  Hopefully it will spark some conversations from the developers, IA (Information Assurance) folks and other like minded people on what they would like to see in an application log.
 
A source that addressed this question directly was the AppSec FAQ at OWASP:
 
  • Do I need to have logging in my application even if I've W3C logs?

Yes, it's important that your application maintains "application level" logs even when W3C logging is used. As W3C logs contain records for every http request, it is difficult (and, at times impossible) to extract a higher level meaning from these logs. For instance, the W3C logs are cumbersome to identify a specific session of user and the activities that the user performed. It's better that the application keeps a trail of important activities, rather than decode it from W3C logs.

  • What should I log from within my application?

Keep an audit trail of activity that you might want to review while troubleshooting or conducting forensic analysis. Please note that it is inadvisable to keep sensitive business information itself in these logs, as administrators have access to these logs for troubleshooting. Activities commonly kept track of are:

  • Login and logout of users
  • Critical transactions (e.g.. fund transfer across accounts)
  • Failed login attempts
  • Account lockouts
  • Violation of policies
The data that is logged for each of these activities usually include:
  • User ID
  • Time stamp
  • Source IP
  • Error codes, if any
  • Priority
 
Mike Gunderloy, in his book "Coder to Developer" (excellent book!, go buy it now!), also devotes an entire chapter to logging application activity. As compared to OWASP, he looks at it more from an application troubleshooting perspective and advises that as a rule, you should "Log the information that you use when debugging".  In particular he recommends looking at logging the following items:
  • Error messages and information, including a stack trace of the error
  • The state of internal data structures at key points in the application
  • User actions, such as button clicks and menu item selections
  • The time the significant actions were performed
  • Important information about the environment, such as variable settings and available memory
The PAG folks in "Improving Web Application Security" recommend the following when it comes to Auditing and Logging:
  • Audit and log access across application tiers
  • Consider identity flow
  • Log key events
  • Secure log files
  • Back up and analyze log files regularly
Michael Howard in "Writing Secure Code 2" has the following suggestions:
  • Log IP Addresses
  • If you are creating a lot of files, you should have your own log files and not use the OS Application Log
  • Logs should go into a directory that is user configurable and it's best to create a new log file every day
  • Consider creating multiple log files, one for routine events another for extraordinary events
  • Application logs should be writable only by the administrator and the user the service runs under
  • When code fails for security reasons, log the data in a place that only the administrator has access to
One surprise in my research was that this topic was almost completely (beyond a couple of sentences) ignored in "Code Complete 2".
 
Comments?