.. _Parser: Parser modules ============== Base64 ------ This module will let you en/decode base64 data. | **source_fields**: Input fields to split. Can be a single field or a list of fields. | **target_fields**: event field to be filled with the new data. Configuration template: :: - parser.Line: action: # source_field: # target_field: # keep_original: # receivers: - NextModule Collectd -------- Parse collectd binary protocol data. This module can receive binary data from the collectd network plugin. Decode: It will parse the collectd binary data and create or replace fields in the internal data dictionary with the corresponding collectd data. Encode: Encode selected fields or all to collectd binary protocol. Configuration template: :: - parser.Collectd: action: # source_fields: # target_field: # keep_original: # receivers: - NextModule Csv --- Parse a string as csv data. It will parse the csv and create or replace fields in the internal data dictionary with the corresponding csv fields. | **source_field**: Field that contains the csv data. | **escapechar**: Char used to escape special characters. | **skipinitialspace**: When True, whitespace immediately following the delimiter is ignored. The default is False. | **quotechar**: A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters. | **delimiter**: A one-character string used to separate fields. | **fieldnames**: Fieldnames to be used for the extracted csv data. Configuration template: :: - parser.Csv: source_field: # escapechar: # skipinitialspace: # quotechar: # delimiter: # fieldnames: # receivers: - NextModule DateTime -------- Parse a string to a time object an back again. For date patterns see: https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior Configuration template: :: - parser.DateTime: source_field: # source_date_pattern: # source_timezone: # target_field: # target_date_pattern: # target_timezone: # receivers: - NextModule DomainName ---------- Parse fqdn to top level domain and subdomain parts. A string like: "http://some.subdomain.google.co.uk" will produce this dictionary: 'domain_name_info': { 'tld': 'google.co.uk', 'domain': 'google', 'suffix': 'co.uk', 'subdomain': 'some.subdomain' } | **source_field**: Input field to parse. | **target_field**: Field to update with parsed info fields. Configuration template: :: - parser.DomainName: source_field: # target_field: # receivers: - NextModule Inflate ------- Inflate any field with supported compression codecs. It will take the source fields and decompress them with the configured codecs. At the moment only gzip an zlib are supported. | **source_fields**: Single field or list of fields to decompress. | **target_fields**: Single field or list of fields to fill with decompressed data. | If not provided, contents of source_fields will be replaced. | **compression**: Compression lib to use for decompression. Configuration template: :: - parser.Inflate: source_fields: # target_fields: # compression: # receivers: - NextModule Json ---- Json codec. Decode: It will parse the json data in source fields and create or replace fields in the internal data dictionary with the corresponding json fields. Encode: It will build a new list of source fields and create json of this list. |**action**: Either encode or decode data. |**source_fields**: Input fields for de/encode. If encoding, you can set this field to 'all' to encode the complete event dict. |**target_field**: Target field for de/encode result. If decoding and target is not set, the event dict itself will be updated with decoded fields. |**keep_original**: Switch to keep or drop the original fields used in de/encoding from the event dict. Configuration template: :: - parser.Json: action: # source_fields: # target_field: # keep_original: # receivers: - NextModule Line ---- Line parser. Decode: Will split the data in source fields and emit parts as new events. So if e.g. data field contains: message-a|message-b|message-c you can split this field by "|" and three new events will be created with message-a, message-b and message-c as payload. The original event will be discarded. | **source_field**: Input field to split. | **seperator**: Char used as line seperator. | **target_field**: event field to be filled with the new data. Configuration template: :: - parser.Line: source_field: # seperator: # target_field: # keep_original: # receivers: - NextModule MsgPack ------- Decode: It will parse the msgpack data and create or replace fields in the internal data dictionary with the corresponding json fields. Encode: Encode selected fields or all to msgpack format. Configuration template: :: - parser.MsgPack: action: # mode: # source_fields: # target_field: # keep_original: # receivers: - NextModule NetFlow ------- Netflow parser Decode netflow packets. | **source_field**: Input field to decode. | **target_field**: Event field to be filled with the new data. Configuration template: :: - parser.NetFlow: source_field: # target_field: # keep_original: # receivers: - NextModule Regex ----- Parse a string by named regular expressions. If regex matches, fields in the data dictionary will be set as defined in the named regular expression. Additionally the field "lumbermill.event_type" will be set containing the name of the regex. In the example below this would be "httpd_access_log". It is also possible to define multiple regexes with the same name. This allows for different log patterns for the same log type, e.g. apache access logs and nginx access logs. | **source_field**: Field to apply the regex to. | **mark_unmatched_as**: Set to this value if regex did not match. | **break_on_match**: Stop applying regex patterns after first match. | **hot_rules_first**: Apply regex patterns based on their hit count. Configuration template: :: - parser.Regex: source_field: # mark_unmatched_as: # break_on_match: # hot_rules_first: # field_extraction_patterns: # - httpd_access_log: ['(?P.*)', 're.MULTILINE | re.DOTALL', 'findall'] receivers: - NextModule SyslogPrival ------------ It will parse the source field in the event dictionary for the default severity and facility fields (RFC5424, http://tools.ietf.org/html/rfc5424). The source field must contain the prival with the pattern: "\d+" Numerical Facility Code 0 kernel messages 1 user-level messages 2 mail system 3 system daemons 4 security/authorization messages 5 messages generated internally by syslogd 6 line printer subsystem 7 network news subsystem 8 UUCP subsystem 9 clock daemon 10 security/authorization messages 11 FTP daemon 12 NTP subsystem 13 log audit 14 log alert 15 clock daemon (note 2) 16 local use 0 (local0) 17 local use 1 (local1) 18 local use 2 (local2) 19 local use 3 (local3) 20 local use 4 (local4) 21 local use 5 (local5) 22 local use 6 (local6) 23 local use 7 (local7) Numerical Severity Code 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages Configuration template: :: - parser.SyslogPrival: source_field: # map_values: False # facility_mappings: # severity_mappings: # receivers: - NextModule Url --- Urlencode or decode an event field and extract url parameters. If no target field is supplied, it will overrite the source field with the parsed date dictionary. | **action**: Either encode or decode data. | **source_field**: Event field to en/decode. | **target_field**: Event field to update with en/decode result. If not set source will be replaced. | **parse_querystring**: Parse url for query parameters and extract them. | **querystring_target_field**: Event field to update with url parameters. | **querystring_prefix**: Prefix string to prepend to url parameter keys. Configuration template: :: - parser.Url: action: # source_field: # target_field: # parse_querystring: # querystring_target_field: # querystring_prefix: # receivers: - NextModule UserAgent --------- Parse http user agent string A string like: "Mozilla/5.0 (Linux; U; Android 2.3.5; en-in; HTC_DesireS_S510e Build/GRJ90) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1" will produce this dictionary: 'user_agent_info': { 'device': { 'family': u'HTC DesireS'}, 'os': { 'family': 'Android', 'major': '2', 'minor': '3', 'patch': '5', 'patch_minor': None}, 'user_agent': { 'family': 'Android', 'major': '2', 'minor': '3', 'patch': '5'}}} | **source_fields**: Input field to parse. | **target_field**: field to update with parsed info fields. Configuration template: :: - parser.UserAgent: source_fields: # target_field: # receivers: - NextModule XPath ----- Parse an xml string via xpath. This module supports the storage of the results in an redis db. If redis-client is set, it will first try to retrieve the result from redis via the key setting. If that fails, it will execute the xpath query and store the result in redis. Configuration template: :: - parser.XPath: source_field: # target_field: # query: # redis_store: # redis_key: # redis_ttl: # receivers: - NextModule