Referer_log and Agent_log for MKStats and Netscape Commerce Server (UNIX)
         Chris J. Gullette (cjgullette@tasc.com), April 5, 1996.
-------------------------------------------------------------------------

In order to get MKStats and Netscape Commerce Server to cooperate with regards to the
agent_log and the referer_log, the following changes must be made (assuming the server
is already set up to record a Common Log File (CLF) -- for details, Read The Fine
Manual or follow the instructions in your Admin Server).  I run by this procedure
on a Sun SPARCstation 20 with Solaris 2.4, and it works just fine:

1)Get Commerce Server to create an agent_log.
---------------------------------------------
Per the Commerce Server Installation and Reference guide:

	A) Add the following line to your obj.conf file:
	   "AddLog fn=record-useragent name=agent_log"

	B) In the magnus.conf file, find the line that starts "Init fn=init-clf..." and
         append " agent_log=**PATH**/agent_log" (where **PATH** is the path name
         to your log files directory.  It will probably be "ns-home/https-80/logs"


2)Get Commerce Server to create a referer_log.
----------------------------------------------
Now this one's a bit more involved:

	A) Put this file, "nosy.c", into the "ns-home/nsapi/examples" directory.

/* NEWNOSY.C -- Chris J. Gullette (cjgullette@tasc.com), April 4, 1996.
 *
 * This code is intended to create a referer log for Netscape Commerce Server
 * in a format usable by the MKStats log analysis program:
 *
 * http://www.referer.com/page.html -> /destination.html/
 *
 * These functions are a modification of "nosy.c", a file published by
 * Netscape Communications Corporation as technical note #20027.  To my
 * knowledge, the original code is not copywrited.  This code was written for
 * a Sun SPARCstation 20 running Solaris 2.4 and using the standard Sun "cc"
 * compiler that comes in the 2.4 bundle.
 */

/* standard headers for SAFs */

#include "base/pblock.h"
#include "base/session.h"
#include "frame/req.h"

/*

  If the DNS-name couldn't be found, the IP address is logged instead;
  if any other field couldn't be found, "-" is logged.

*/

/* specific headers useful for logging functions */

#include "base/file.h"
#include "base/util.h"

/* File descriptor to be shared between the processes */

static SYS_FILE logfd = SYS_ERROR_FD;

/* function to shut down the logging; will be called on server restart */

void nosy_term(void *parameter)
{
    system_fclose(logfd);
    logfd = SYS_ERROR_FD;
  }

/* init function for logging */

int nosy_init(pblock *pb, Session *sn, Request *rq)
{
    /* file parameter */
    char *fn = pblock_findval("file", pb);

    if(!fn) {
        pblock_nvinsert("error", "nosy-init: please supply a file name", pb);
        return REQ_ABORTED;
      }
    logfd = system_fopenWA(fn);
    if(logfd == SYS_ERROR_FD) {
        pblock_nvinsert("error", "nosy-init: please supply a file name", pb);
        return REQ_ABORTED;
      }
    /* Close log file when server is restarted */
    magnus_atrestart(nosy_term, NULL);
    return REQ_PROCEED;

  }

/* actual logging function! */

int nosy_log(pblock *pb, Session *sn, Request *rq)
{
    /* working variables */
    char *uri, *referer, *empty="-\0";
    char *logmsg;
    int len;

    /* stuff we want to log */
    uri = pblock_findval("uri", rq->reqpb);
    referer = pblock_findval("referer", rq->headers);

    /* make NULL results be a dash */

    if (!referer) referer=empty;

    /* print it! */
    logmsg = (char *)
        MALLOC(strlen(referer)+strlen(uri)+7);
    len=util_sprintf(logmsg, "%s -> %s\n", referer, uri);
    system_fwrite_atomic(logfd, logmsg, len);
    FREE(logmsg);

    return REQ_PROCEED;
  }
/* END newnosy.c */

	B) Edit the OBJS line in that directory's Makefile to append "nosy.o":
	   "OBJS = addlog.o auth.o ntrans.o otype.o pcheck.o service.o nosy.o"

	C) Run "make" to make a new examples.so file.

	D) Edit your magnus.conf file:
		1. Add this to the "Init-fn=init-clf..." line: 
               " referer_log=**PATH**/referer_log"
		2. Add these lines:
		   "Init fn=load-modules shlib=/usr/ns-home/nsapi/examples/example.so \
		    funcs=nosy-init,nosy-log
		    Init fn=nosy-init file=/usr/ns-home/https-80/logs/referer_log"

	E) To the "obj.conf" file, add the following line in the
	   "<Object name=default>" section:
	   
		AddLog fn=nosy-log name="referer_log"


3)Get MKStats to process your agent_log correctly:
--------------------------------------------------
Commerce Server's default agent_log is different from those that MKStats is designed
to work with.  It prints the IP address of the referencing site at the beginning of
each line [255.254.253.252 Mozilla/2.0 (X11; I; SunOS 5.4 sun4m)].  To accommodate
this difference, you must make a minor change to the sub.agent script.  Only one
line needs to be changed as follows:

############ START sub.agent ###########
# =====================================================================
# agent: process the agent_log
# =====================================================================
sub agent {

  open(FILE,"$AGENTLOG");
  while (<FILE>) {
    chop;
    (($. % 500)==0) && (print STDERR "\#") if $verbose;
    ($browser,$junk) = split(/[\s]*\(/,$_,2);
# The next line has been modified from the original MKStats script to
# accommodate the unusual agent_log that Commerce Server generates.
    ($junk,$browser) = split(/\s/,$browser,2);
    ($browser,$version) = split('/',$browser);
    $version =~ s/\s*$//;
    if ($version eq '') { $version="none"; }
    $agenttotal{$browser}++;
    $agent{$browser}{$version}++;
    }
  close(FILE);
}

1;
########### END sub.agent #############

Don't forget to make the appropriate changes to the mkstats.config file so that
the program can find the new log files and process them.  Particularly, be sure to
change the $REF_IGNOREDOMAIN variable in your mkstats.config file to 
"www.your.website.com", or you'll score a huge number of in-site references.

4) Restart your Commerce Server:
--------------------------------
That's it!  Once you've made all these changes, clear out your old logs if you so
desire, restart the server, and you should have browser and referer info in MKStats.

Chris Gullette, cjgullette@tasc.com, April 5, 1996.