Will Wingate 2008 have logs that we can open in Excel?

[pgr]

Hi,

still waiting for the new Wingate...

I just wanted to ask if it will have log files with better formats, something we can open in Excel like CSV, or even a tab-separated thing, but one where one line has everything to describe one event (as opposed to events spanning multiple lines - these are a pain to filter), and columns are better organized.

Something so simple as going through a log to find out the largest downloads (byte size) is currently a VERY daunting task.

Since this only a matter of going through a bunch of fprintfs and making formats a bit more uniform, it would really help if it was improved in Wingate 2008.

Thanks!

[adrien]

Hi

the plan is to make log format customizable.

It's not currently implemented in WinGate 2008 log module yet, but actually you can currently (if you had it) log to an ODBC data source from within policy, or log to a file in any format from within a Lua script policy element. Completely your choice as to what you log and what format.

However, along with this we need to ship default / sample config to allow people to use logging without having to design log formats to get off the ground.

We've identified a couple of standard log formats, e.g. csv, W3C, any other requests?

Regards

Adrien

[adrien]

another option if you're into writing code will be the ability to write your own event hooking module.

All log events are (as well as all other types of published events) able to be hooked by external modules.

The context data available with the event is basically all the data winGate has about something. E.g if it's an event associated with a session (e.g a web request event etc), you get the session data, such as client IP, username, HTTP request details etc etc.

All this data is available through an API we've written. So, you get to hook the events, you get all the data. What you then do with it is up to you.

[pgr]

We've identified a couple of standard log formats, e.g. csv, W3C, any other requests?

More than the file format, I'm concerned with the content format.

CSV is fine, as long as the info is nicely laid out. Excel's auto-filter option is my favorite to handle any type of data. It's very easy and powerful.

Having two or three lines for a single proxy request, for example, kills the chances of using auto-filter effectively. But if everything is kept in a one-record-per-line format, with constant column meanings, it should be pretty straight-forward to analyze.

Thanks for your answers!

[DrWho]

Are you aware that current WG logs will open in Excel (2003 and 2007 were tried) in their native format?

Just right click a log file, select 'open with...' and pick Excel. Excel automatically interprets the correct delimeter putting all the data in neat columns. You can the sort by the column of your choice, perform macros, or save in xls format, etc.

James

[pgr]

Yes, I can open it in Excel, and each line gets correctly separated by columns:

Code: Select all

7/08/2008 19:33   10.2.3.4   joe   90695   Requested:   http://forums.qbik.com/styles/prosilver/template/styleswitcher.js            
7/08/2008 19:33   10.2.3.4   joe   90695   Traffic   2743   659   0   0   0s
7/08/2008 19:33   10.2.3.4   joe   90695   Requested:   http://forums.qbik.com/styles/prosilver/template/forum_fn.js            
7/08/2008 19:33   10.2.3.4   joe   90695   Traffic   4601   654   0   0   0s
7/08/2008 19:33   10.2.3.4   joe   90695   Requested:   http://forums.qbik.com/styles/prosilver/theme/print.css            
7/08/2008 19:33   10.2.3.4   joe   90695   Traffic   117   701   0   0   0s
7/08/2008 19:33   10.2.3.4   joe   90696   Requested:   http://forums.qbik.com/style.php            
7/08/2008 19:33   10.2.3.4   joe   90697   Requested:   http://forums.qbik.com/styles/prosilver/imageset/qbik_logo.gif            
7/08/2008 19:33   10.2.3.4   joe   90698   Requested:   http://forums.qbik.com/styles/prosilver/imageset/icon_post_target.gif            
7/08/2008 19:33   10.2.3.4   joe   90698   Traffic   117   731   0   0   0s
7/08/2008 19:33   10.2.3.4   joe   90697   Traffic   2518   686   0   0   0s


But as you can see from the above sample from WWW Proxy log, the format across lines is not constant, and the info regarding a single request is spread out across several lines, possibly interrupted by other information.

This makes it complicated to gather information about a request, sort by largest/longest downloads, find request with errors etc.

I know that Wingate is just spitting out the info as it happens, but I would much prefer if it would gather information about a request and put everything together in a line, keeping columns constant:

timestamp 10.2.3.4 joe Request http://something - - - -
timestamp 10.2.3.4 joe Traffic http://something 1234 212 0 0 1s
timestamp 10.2.3.4 joe Error http://something 1234 212 0 0 1s Errornum

The information would build up, but I can easily filter for a single line and see everything there, in the expected column.

Thanks.

[logan]

In WinGate 2008, you can make WinGate record log files in any format that you require, so it will definitely be possible to have your logs formatted like this.