Tenon Intersystems : WebTen White Papers

WebTen Logging

Before you can analyze your log files, you need to understand what kind of data is actually recorded there, as well as its format. The most commonly used log-file format is called the CLF (Common Log Format). Most Web servers offer CLF at least as an option, if not the default. Each line in a CLF log file corresponds to a single request made by the user to the Web server. The format specifies data such as the machine name or IP address from which the user made the request, as well as the information on the resource requested.

WebTen can be configured to have either Apache or Squid do its logging. When using Apache, WebTen for Mac OS supports two customizable logging formats: The CLF (Common Log Format), which also includes the Extended Log Format, and the WebSTAR log format. iTools for Mac OS X does not support the WebSTAR log format. When WebTen uses the Squid cache, Squid does all the logging in CLF (Common Log Format). The advantage to using Squid cache is that your server will run significantly faster, but at the expense of a less flexible log file format. If you are using Squid, see the section below on Squid logging.

The Apache-based logging is controlled via the Administration Server (http://servername/webten_admin). If you decide to use Apache logging you must start by turning off the Squid cache. To do this, click on the Cache button to get to the Cache Settings screen. From the Cache Setting screen click both the AcceleratorCache and the Supercache Enable toggle switches to the off position. Squid is only turned off when you click Save Cache Settings. Once Squid is turned off, return to the home page. Click on the Server Defaults button to apply your configuration to the whole site, or click on each individual Virtual Host Config button to apply your configuration to that individual site.

Once on the configuration page, you will notice that there are four log Directives with corresponding Value columns. The ErrorLog, TransferLog, and ScriptLog entries are used to set the file folder and file name, all of which start their path from the WebSites folder and use forward slash to separate folders. If you don't put a file name in the TransferLog Directive, then Apache will only write the logging to the disk if the Display Access Log window is opened. The LogFormat Directive is the place to custom configure the type of data your TransferLog file records. A toggle switch will let you choose between the standard WebSTAR Log format and the default standard CLF (Common Log Format). In the Value column you can customize your own log format rather than using one of the standard formats (note that if you use the toggle switch, then the Admin Server will replace any custom format symbols inserted with one of the standard configurations).

To create a custom format, insert your choice of Format Symbols, with a percent sign before each symbol. For example "%h %l %u %t %r %b" would be a functional format setting. Don't forget the quotation marks.




 Log File Format Command Symbol Definitions: 

 %W (Log records in Webstar format)
 %h (The hostname of the client, (or IP number if hostname is not available or if 
     DNSLookup is off.))
 %u (remote user from authorization, if any.)
 %t (date and time in CLF format: (day/month/year:hour:minute:second zone.)) 
 %r (first line of request exactly as it came from the client (i.e., the file
     name, and protocol requested.))
 %s (original http request status code returned to client before internal 
     redirection.  Indicates where or not the file was successfully retrieved, 
     and if not, what error message was returned.)
 %>s (final http request status code.)
 %b (number of bytes sent, not including headers.)
 %U (url path requested)
 %T (transfer time or time taken to serve a request in seconds)
 %p (TCP port of the server servicing the request)
 %P (process ID of the server servicing the request)
 %l (the clients remote logname, if supplied)
 %v (name of (virtual) server servicing request)
 %w (WebSTAR result, i.e. "OK", "ERR", or "PRIV")
 %d (date and time in WebSTAR format)
 %{}n (contents of note from another module in brackets)
 %{}i (Input header item in brackets)
 %{}o (output header item)
 %{Referer}i  (The URL the client was on before requesting your URL.
 %{User-agent}i  (The identity of the client software (browser.))

Notice in the above list, the input and output header commands.  The 
referer and user-agent commands are discribed in the list of commonly 
used HTTP Header Fields in the next section.

 If you are running iTools (which uses apache 1.3), then these additional 
commands are available
 
%f (Filename)
%a (Remote IP address)
%V (The server name according to the UseCanonicalName setting.)

  Some Configuration Examples: 

Standard WebSTAR Format
DATE TIME RESULT HOSTNAME URL BYTES_SENT

"%W %d %w %h %>U %b"

Standard Common Log Format (CLF) (Default)
HOSTNAME LOGINNAME USER GMT_TIME "REQUEST" STATUS BYTES_SENT

"%h %l %u %t \"%r\" %>s %b"

WebSTAR Custom Format Example
HOSTNAME USER_AGENT USER GMT_TIME 'REQUEST' RESULT STATUS BYTES_SENT

"%W %h %{User-Agent}i %>u %t '%r' %w %>s %b"

Squid Logging

Like the Apache logging, if you don't put a file name and path in the TransferLog Directive on the (Server Defaults) screen, then Squid will only write the logging to the disk if the Display Access Log window is open. The default Squid log format is:

	 remotehost rfc931 authuser [date] "method URL" status bytes

This logging configuration can not be changed, but you can add HTTP Header Fields. If you edit the webten/tenon/squid/etc/Squid.conf file using a text editor such as BBedit, at the very end of the file, you will notice the lines:

#  TAG: log_http_hdrs
#	Append individual HTTP request headers to CLF log entry

# log_http_hdrs Referer User-Agent

These are the commented enteries. The pound sign indicates that these lines are comments only. The last line is an example implementing the most popular HTTP Header Fields, Referer and User-Agent. Using the same format, you can add any Header Field you want. Or you can remove the # on the last line and log these example Header Fields. Using these example Header Fields would be the configuration needed to process your log files via Funnel Web.

If you need more information on understanding the information in the log files or the configuration of Squid, then check Squid Log Files. If you need to Roll your log file or split it in to multiple files by host, then check out our files posted online.

List of commonly used HTTP Header Fields

You can use any HTTP Header Field, this is just a list of the most commonly used with definitions. For a complete list, check Hypertext Transfer Protocol -- HTTP/1.1.

HTTP_ACCEPT The MIME types which the client will accept, as given by HTTP headers. Other protocols may need to get this information from elsewhere. Each item in this list should be separated by commas as per the HTTP spec.
Format: type/subtype, type/subtype
HTTP_USER_AGENT
The browser the client is using to send the request. General format: software/version library/version.

The server sends back to the client:

A status code that indicates whether the request was successful or not. Typical error codes indicate that the requested file was not found, that the request was malformed, or that authentication is required to access the file.
The data itself. Since HTTP is liberal about sending documents of any format, it is ideal for transmitting multimedia such as graphics, audio, and video files.
It also sends back information about the object being returned.

Content-Type

Indicates the media type of the data sent to the recipient or, in the case of the HEAD method, the media type that would have been sent had the request been a GET.

           Content-Type: text/html

Date

The date and time at which the message was originated.

           Date: Tue, 15 Nov 1994 08:12:31 GMT

Expires

The date after which the information in the document ceases to be valid. Caching clients, including proxies, must not cache this copy of the resource beyond the date given, unless its status has been updated by a later check of the origin server.

           Expires: Thu, 01 Dec 1994 16:00:00 GMT

From

An Internet e-mail address for the human user who controls the requesting user agent.

           From: Stars@WDVL.com

The request is being performed on behalf of the person given, who accepts responsibility for the method performed. Robot agents should include this header so that the person responsible for running the robot can be contacted if problems occur on the receiving end.

If-Modified-Since

Used with the GET method to make it conditional: if the requested resource has not been modified since the time specified in this field, a copy of the resource will not be returned from the server; instead, a 304 (not modified) response will be returned without any data.

           If-Modified-Since: Sat, 29 Oct 1994 19:43:31 GMT

Last-Modified

Indicates the date and time at which the sender believes the resource was last modified. Useful for clients that eliminate unnecessary transfers by using caching.

           Last-Modified: Tue, 15 Nov 1994 12:45:26 GMT

Location

The Location response header field defines the exact location of the resource that was identified by the request URI. If the value is a full URL, the server returns a "redirect" to the client to retrieve the specified object directly.

Location: http://WWW.Stars.com/Tutorial/HTTP/index.html

If you want to reference another file on your own server, you should output a partial URL, such as the following:

Location: /Tutorial/HTTP/index.html

Referer

Allows the client to specify, for the server's benefit, the address (URI) of the resource from which the request URI was obtained. This allows a server to generate lists of back-links to resources for interest, logging, optimized caching, etc. It also allows obsolete or mistyped links to be traced for maintenance.

           Referer: http://WWW.Stars.com/index.html

Server

The Server response header field contains information about the software used by the origin server to handle the request.

           Server: CERN/3.0 libwww/2.17

User-Agent

Information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations - such as the inability to support HTML tables.

           User-Agent: CERN-LineMode/2.15 libwww/2.17b3

If you don't like to look at log files like this one:

Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4239 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:demo1_l.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:29:49 00:00:00 4 :staff:Raul:psx:pics:demo1_l.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 1893 1893 4236 www.brownshipley.com. - GET - :cgi:counter:Counter.acgi 12/31/96 www.brownshipley.com. GET count_psx/Default http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:29:49 00:00:02 160 :cgi:counter:Counter.acgi Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4241 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:dd_l.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:29:50 00:00:00 54 :staff:Raul:psx:pics:dd_l.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4242 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:psx.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:29:51 00:00:00 39 :staff:Raul:psx:pics:psx.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4244 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:wipeout_l.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:29:51 00:00:00 53 :staff:Raul:psx:pics:wipeout_l.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4243 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:psx_board.jpg 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:29:52 00:00:00 48 :staff:Raul:psx:pics:psx_board.jpg Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4247 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx.html 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://av.yahoo.com/bin/search?p=%2BPSX+AND+%2BPIRATE OK 21:33:07 00:00:00 38 :staff:Raul:psx.html Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4245 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:w:PALBACK.jpg 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:33:09 00:00:01 66 :staff:Raul:psx:pics:w:PALBACK.jpg Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4251 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:psx_l.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:33:10 00:00:01 74 :staff:Raul:psx:pics:psx_l.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4253 www.brownshipley.com. - CONDITIONAL_GET 304 :netnow3.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:33:10 00:00:00 48 :netnow3.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4252 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:dot.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:33:12 00:00:00 5 :staff:Raul:dot.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4248 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:demo1_l.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:33:12 00:00:00 8 :staff:Raul:psx:pics:demo1_l.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4249 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:dd_l.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:33:12 00:00:00 12 :staff:Raul:psx:pics:dd_l.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 1942 1942 4246 www.brownshipley.com. - GET - :cgi:counter:Counter.acgi 12/31/96 www.brownshipley.com. GET count_psx/Default http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:33:12 00:00:03 206 :cgi:counter:Counter.acgi Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4256 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:psx.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:33:14 00:00:00 41 :staff:Raul:psx:pics:psx.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4250 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:wipeout_l.gif 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:33:14 00:00:01 61 :staff:Raul:psx:pics:wipeout_l.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 0 0 4254 www.brownshipley.com. - CONDITIONAL_GET 304 :staff:Raul:psx:pics:psx_board.jpg 12/31/96 www.brownshipley.com. CONDITIONAL_GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:33:14 00:00:01 62 :staff:Raul:psx:pics:psx_board.jpg Mozilla/0.96 Beta (X11; Linux 1.1.59 i486) 1578 1578 4255 rbkc.demon.co.uk. - GET 200 :staff:raul:psx:psxchip.html 12/31/96 rbkc.demon.co.uk. GET http://av.yahoo.com/bin/search?p=psx+cdr OK 21:35:37 00:00:00 9 :staff:raul:psx:psxchip.html Mozilla/0.96 Beta (X11; Linux 1.1.59 i486) 22702 22702 4260 rbkc.demon.co.uk. - GET 200 :staff:raul:psx:pics:18pin-80QFP1000.jpg 12/31/96 rbkc.demon.co.uk. GET http://www.medfac.unimelb.edu.au/staff/raul/psx/psxchip.html OK 21:35:41 00:00:01 94 :staff:raul:psx:pics:18pin-80QFP1000.jpg Mozilla/0.96 Beta (X11; Linux 1.1.59 i486) 23027 23027 4261 rbkc.demon.co.uk. - GET 200 :staff:raul:psx:pics:18pin-80QFP1002.jpg 12/31/96 rbkc.demon.co.uk. GET http://www.medfac.unimelb.edu.au/staff/raul/psx/psxchip.html OK 21:35:41 00:00:00 11 :staff:raul:psx:pics:18pin-80QFP1002.jpg Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 9588 9588 4263 www.brownshipley.com. - GET 200 :staff:Raul:psx.html 12/31/96 www.brownshipley.com. GET http://av.yahoo.com/bin/search?p=%2BPSX+AND+%2BPIRATE OK 21:35:57 00:00:00 40 :staff:Raul:psx.html Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 4376 4376 4264 www.brownshipley.com. - GET 200 :staff:Raul:psx:pics:w:PALBACK.jpg 12/31/96 www.brownshipley.com. GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:36:00 00:00:01 71 :staff:Raul:psx:pics:w:PALBACK.jpg Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 2847 2847 4265 www.brownshipley.com. - GET 200 :staff:Raul:psx:pics:psx_l.gif 12/31/96 www.brownshipley.com. GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:36:02 00:00:02 150 :staff:Raul:psx:pics:psx_l.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 1943 1943 4268 www.brownshipley.com. - GET - :cgi:counter:Counter.acgi 12/31/96 www.brownshipley.com. GET count_psx/Default http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:36:02 00:00:02 145 :cgi:counter:Counter.acgi Mozilla/0.96 Beta (X11; Linux 1.1.59 i486) 4215 4215 4262 rbkc.demon.co.uk. - GET 200 :staff:raul:psx:chip_hack.html 12/31/96 rbkc.demon.co.uk. GET http://www.medfac.unimelb.edu.au/staff/raul/psx/psxchip.html OK 21:36:03 00:00:07 450 :staff:raul:psx:chip_hack.html Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 1884 1884 4266 www.brownshipley.com. - GET 200 :netnow3.gif 12/31/96 www.brownshipley.com. GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:36:05 00:00:00 40 :netnow3.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 1115 1115 4267 www.brownshipley.com. - GET 200 :staff:Raul:dot.gif 12/31/96 www.brownshipley.com. GET http://www.medfac.unimelb.edu.au/staff/Raul/psx.html OK 21:36:05 00:00:00 31 :staff:Raul:dot.gif Mozilla/2.0 (X11; I; Linux 1.2.3 i486) 5322 5322 4269 www.brownshipley.com.

Then use one of the many log viewers out on the market.

WebTen Logging

Log File Format Command Symbol Definitions:

If you are running iTools (which uses apache 1.3), then these additional commands are available

Some Configuration Examples:

Squid Logging

List of commonly used HTTP Header Fields

If you don't like to look at log files like this one:

Then use one of the many log viewers out on the market.