This paper will discuss the "ins" and "outs" of iTools logging. It covers the logging capabilities of both Squid and Apache, and how they work in relation to each other. Two different, yet powerful, methods of log rolling and splitting are discussed also.
The design philosophy with iTools was to put all of our enhancements in place of the default OS X Server Apache. As a result, we do not edit the /Local/Library/WebServer/Configuration/apache.conf file upon installation of iTools. As a result of this, the default Apache directives are left in place after iTools is installed. This appears to the user to be quite confusing at first. The LogFormat and CustomLog directives control the access log in Apache. The directives as delivered by Apple are as follows:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog "|/usr/sbin/rotatelogs /Local/Library/WebServer/Logs/apache_access_log 86400" c ommon
All hits that go to Apache are logged in the "common" format. If you wanted the Referer and User-Agent to be logged, then you could change the CustomLog directive to say "combined" instead of common.
Note that the above reference to "CustomLog" logs hits to Apache. Under iTools, this means something different than it did under Apple's stock Apache. Since iTools makes use of the Squid object cache, Apache does not see all of the hits that the server receives. Most of the hits to a properly-configured Squid cache will be served by Squid itself and Apache will be left to field requests to CGI scripts, Apache modules, and other such dynamic web pages. Since the apache_access_log file doesn't receive all of the hits under iTools, it is rather useless. The recommended step would be to comment out the "CustomLog" directive in your apache.conf file to save your system the overhead and memory usage.
The default iTools log is called iTools.log, and it is also located in the /Local/Library/WebServer/Logs directory along with the apache_access_log. The logging to this file is controlled by the "CacheTransferLog" directive in the apache.conf file and the cache_access_log directive in the /usr/local/squid/etc/squid.conf file.
It is recommended that you use the Squid cache as it significantly improves the speed of your server and can offer virtually the same functionality as Apache's logs.
Squid has the ability to log the Referer and User-Agent in its Common Log Format just like Apache can. In iTools this ability is disabled by default, but it can be easily enabled. Simply edit the squid.conf file and change the line:
#log_http_hdrs Referer User-Agent
TO:
log_http_hdrs Referer User-Agent
You will have to restart the server after this is done by typing "apachectl restart" on the command line. Your iTools.log will now display the Referer and the User-Agent.
Because of the use of the Squid cache, the default iTools installation does not include the log rolling features that Apple's original Apache did. Fortunately, log rolling in iTools can be made far more flexible and powerful.
A common task that web administrators need to accomplish is to split up the logs into separate files so that they can be either processed by a program such as Funnel Web Pro, and/or placed into the virtual host folders so that virtual-host site administrators can view the log files.
We have written a script to do just this. This script takes the main iTools.log file, archives it in the Logs/archive folder, and splits it into separate files for each virtual host. The split logs are put into a "logs" folder inside each virtual host folder.
Installing the script involves simply following the instructions given in the readme file that accompanies the script (which includes editing the crontab file for automatic log rolling).
Please note that extracting this archive will give you a folder called "log_splitter". This folder must be placed in the "/Local/Library/WebServer" directory. Instructions indicating this are present in the readme file, but the script will not work if it is not placed here. Just a reminder.
(The following instructions are based on a message posted to the iTools mailing list from Todd Daniel Woodward who is a technical consultant for http://www.radiodigest.com)
This log rolling method uses Squid's internal log rolling features to roll the logs. This may be more convenient if you do not want your logs split up into your separate virtual hosts. The first step involves editing the /usr/local/squid/etc/squid.conf file.
Under the "LOGFILE PATHNAMES AND CACHE DIRECTORIES" section in the squid.conf file, change "cache_access_log" to the directory and the name you want your logs to be. Example: We have two servers, so for Web01 we do "cache_access_log /Local/Library/WebServer/Logs/web01_apache.log".
Under the "MISCELLANEOUS" section in the squid.conf file, change "logfile_rotate" to the number of log files you want to rotate through. If you want to rotate through a weeks worth of logs, then it would be "logfile_rotate 10".
You'll need to create a cron job to tell Squid to rotate it's logs when you want it to. cron executes programs, scripts, whatever, at intervals you specify. (Seconds, Minutes, Days, Weeks, Months...you get the picture.)
At the command line, type "crontab -l". This will show you a list of all cron jobs you have programmed, if you have any. This command is not necessary, it's just to demonstrate what you're going to be working with.
Add the following line to your crontab file:
0 0 * * * /usr/local/squid/bin/squid -k rotate
cron will execute the "squid -k rotate" command at zero hour (midnight) to run Squid with the command line argument "-k rotate". Squid will close the current log, rename it with a ".#" at the end, and then create a new log file. Each successive day the log files are rotate and given a higher number.
Example: If our log file is named "podunk_apache.log", then at midnight Squid would rotate this to "podunk_apache.log.0". The next day the ".0" log file would be renamed with a ".1" at the end, and the current ".log" file would be renamed ".log.0". Your most current rotated log file will always have ".0" at the end.
Some users may find that the convenience, versatility, and flexibility of Apache's logging far outways the speed advantages of Squid's logging capabilities. On a site with a large amount of dynamic content (and few images), it is possible not to see a significant decrease in speed when Squid is turned off. In these situations, it may be desireable to put Apache's powerful logging capabilities to use.
Below is the snippet from the apache.conf file that we have seen before.
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog "|/usr/sbin/rotatelogs /Local/Library/WebServer/Logs/apache_access_log 86400" c ommon
Instead of recommending that you comment out the "CustomLog" directive, this section will inform you of how to use it. Please note that you should have already disabled the Squid cache in the "Cache" section of the iTools Admin server.
The "LogFormat" directives define the format of your log files. The last word on the "LogFormat" line is a label. This label can be used to assign a log format to a log file. For example, in the above example, the apache_access_log uses the "combined" format which includes "Referer" and "User-Agent" log directives. Feel free to create your own "LogFormat" directives with their own labels to further customize your logs.
The following table describes LogFormat parameters and a description of each. This information may be useful when putting together a customized LogFormat directive.
| LogFormat parameter | Description |
| %h | The hostname of the client (or IP address if the hostname is not available or if DNSLookup is off). |
| %u | Remote user from authorization (if any). |
| %t | Date and time in CLF format. (day/month/year:hour:minute:second zone) |
| %r | First line of request exactly as it came from the client (i.e. the file name and protocol requested). |
| %s | Original http request status code returned to the client before internal redirection. Indicates whether or not the file was successfully retrieved, and if not, what error message was returned. |
| %>s | Final http request status code. |
| %b | Number of bytes sent. (Not including headers). |
| %U | URL path requested. |
| %T | Transfer time or time taken to serve a request in seconds. |
| %p | TCP port of the server servicing the request. |
| %P | Process ID of the server servicing the request. |
| %l | The client's remote logname, if supplied. |
| %v | Name of (virtual) server servicing the request. |
| %{}n | Contents of note from another module in brackets. |
| %{}i | Input header item in brackets. |
| %{}o | Output header item. |
| %{Referer}i | The URL the client visited before requesting your URL. |
| %{User-agent}i | The identity of the client software (browser). |
| %f | Filename requested. |
| %a | Remote IP address. |
| %V | The server name according to the "UserCanonicalName" setting. |
Apache log rolling is controlled by modifying the numeric value in the following directive.
CustomLog "|/usr/sbin/rotatelogs /Local/Library/WebServer/Logs/apache_access_log 86400" c ommon
The numeric value (in this case 86400) represents the number of seconds that Apache will keep its logs before it rolls them. The default value (86400) represents the number of seconds in a day. You may want to increase or decrease this value depending on how much traffic your site deals with and how large you want your logs to be. On a site with less traffic, you may want to increase the value to 604800 which is the number of seconds in a week. On sites with more traffic, you may want to decrease the value to 43200 which will roll your logs every twelve hours.
Remember to restart your server by typing "apachectl restart" after making any changes to the Apache config files.
Here we have presented two methods to manage your log files, and a detailed description of how logging works in iTools. If you want your logs split and placed in virtual host folders, then choose option 1. If you just want your logs rolled, choose option 2. If you are good at Perl, you can modify our log splitting script to meet your needs.
This paper has also presented a "quick start" guide to Apache-only logging when Squid is disabled. These three methods of logging should help to clarify any logging-related issues you may have experienced with iTools.
If you have any trouble with either method, please contact the Tenon Technical Support Department.
Page last updated May 3, 2000
http://www.tenon.com/support/itools/papers/itools-logging.html