Configuration file for a generic log file parser

Question

Our company uses multiple log file formats.

We would like to develop a series of tools to parse them, often the same core functionality for multiple log file formats.

A classical example is generating Message Sequence Charts from the log files (other candidates are memory usage, stack size and time measurement between events).

To stick with Message Sequence Charts, we basically need to identify:

• the title of the columns (i.e processes involved)
• the title of the messages
• perhaps also message parameters

These, we intend to determine by parsing the log files.

However, for different formats, we might say that the information is

• delimited by certain strings (<some text> <process name> SEND <message> TO <process name>)
• in fixed column positions
• in the Nth word of lines beginning with a certain string
• and so on

This could turn into a spaghetti of command line arguments, so we are thinking of an external configuration file.

What sort of file (plain text, .INI, XMl, or what) and how should we structure it?

I would hope that about 90% of the core of our scripts is common, and we just need 10% or so to massage the input to direct our parsing.

Any advice, or references?

Answer 1

Do not reinvent the wheel, use something like AWK. So your "configuration file" is simply an awk script, which is powerful enough to deal with all the requirements you mentioned.

If that turns out to be not powerful enough, use your favorite scripting language like Perl or Python. And if you can live with a closed source Windows-only solution (freeware, but not FOSS), specificially designed for processing log files, you can try Microsoft's log parser.