Text Log Parsing Schemas

GFI EventsManager supports parsing of Text Logs, based on regular expressions. This enables you to manually define a set of expressions to parse Text Log events, such as W3C Logs (Webserver), CSV Logs or custom logs. This set of expressions is called a Parsing Schema. The Text Log parsing schema is stored in %GFI EventsManager install directory%\Data\toolcfg_textlogParseSchemas.xml. All schemas are stored in the ArrayOfTextLogParseSchema tag in this XML file, and for each schema, there is a TextLogParseSchema, as described below.

Editing this file requires knowledge of XML and regular database expressions. Use the below schema as a guide for structuring your XML file. The elements are described below.

Text Logs Parse Schemas – XML Structure

<TextLogParseSchema>

<Name>ESM Logs(sample)</Name>

<Type>Custom</Type>

<Fields>

<Field>

<Name>date str</Name>

<GroupName>date_str</GroupName>

<Type>String</Type>

</Field>

<Field>

<Name>time str</Name>

<GroupName>time_str</GroupName>

<Type>String</Type>

</Field>

<Field>

<Name>miliseconds</Name>

<GroupName>miliseconds</GroupName>

<Type>String</Type>

</Field>

<Field>

<Name>process id</Name>

<GroupName>process_id</GroupName>

<Type>String</Type>

</Field>

<Field>

<Name>method</Name>

<GroupName>method</GroupName>

<Type>String</Type>

</Field>

<Field>

<Name>description</Name>

<GroupName>description</GroupName>

<Type>String</Type>

</Field>

</Fields>

<Formats>

<Format>

<FileMask>*.csv</FileMask>

<Regex><![CDATA[(?<date_str>\d{4}-\d{2}-\d{2}),\s(?<time_str>\d{2}:\d{2}:\d{2}),\s(?<miliseconds>\d{3}),\s(? <process_id>\w+),\s(?<thread_id>\w+),\s(?<event_type>\w+),\s(?<source>.+),\s(?<method>.+),\s(?<description>.+),\s]]></Regex>

</Format>

</Formats>

<HasHeader>false</HasHeader>

<IgnoreHeader>true</IgnoreHeader>

<HeaderLineCount>2</HeaderLineCount>

<HeaderRegEx />

<HeaderSeparator>,</HeaderSeparator>

<ConcatenateLastValue>true</ConcatenateLastValue>

<LineSeparator>,</LineSeparator>

<DiscardInvalidLines>true</DiscardInvalidLines>

</TextLogParseSchema>

Schema parsing elements

Text Log Parsing Schema	XML Tag Description
Name	The display name of the schema. The provided name is displayed in GFI EventsManager Management Console user interface.
Type	Represents the type of the schema. Supported values for this node include W3C, CSV, XML, DHCP logs, SAP logs, SKI Data logs as well as Custom text logs; example: MySchema. Fields and Formats nodes are unavailable for W3C and CSV.
Formats	A collection of Format nodes, each containing a regular expression for parsing. Format nodes contain: A regular expression for parsing the message and a mask to test files. The mask is in standard command line format and uses “?” and “*” as wildcard characters A standard .net regex element used for parsing expressions. For more information, refer to http://msdn.microsoft.com/en-us/library/az24scfc%28v=vs.71%29.aspx.
Fields	A collection of Field nodes, each containing information about: Name – the display name of the field. The provided name appears in Events Browser GroupName – represents the group name from a regular expression corresponding to the field. This is used to get the field value after parsing Type – data type of the field. Example: String and Date.
IgnoreHeader	Ignores the column header if the value is True. Otherwise, False.
HeaderLineCount	Specify the number of lines in the column header row. Example: <HeaderLineCount>2</HeaderLineCount>
HeaderRegex	Regular expression for parsing the header. Example: <HeaderRegEx />
HeaderSeparator	When no regex is present, specify the separator used to parse fields in the header. Example: <HeaderSeparator>,</HeaderSeparator>
ConcatenateLast Value	Concatenates remaining values to the last field value. Example: if a line is parsed into ten fields and there are only eight header fields, field nine and ten are concatenated to field eight.
LineSeparator	Specify a character for line separation. Example: <LineSeparator>/</LineSeparator>
DiscardInvalidLines	Discards lines that cannot be parsed if the value is True. Otherwise, False.

XML Document Map

XML Document Map