Recently I found myself in the unhappy position of needing to sift through slightly more than a billion Checkpoint Firewall-1 log lines, looking for specific patterns of access. The problem was that many of the exported fwm log files had differing column positions and there had been many ruleset changes over the course of 11 months worth of log data. Many of the excellent FW1 log summarization tools (such as Peter Sundstrom’s fwlogsum) didn’t handle the hundreds of files and differing column positions.

The final scripted solution was processing over 11,000 lines/second .. and still took over 23 hours for the first run.

Log file exports via fwm logexport can have variable column positioning, except for record ID number “num”, which is *always* column number one.  I see three viable alternatives to the changing column position in the ASCII log files exported via fwm – so we can automate the log processing:

1. Parse the header line (line #1) of every log file and dynamically map (rearrange) the columns to a pre-determined standard in memory before further processing (painful, expensive)
2. Tell Checkpoint fwm to export in a fixed column ordering
   create
   logexport.ini
   and place in
   $FWDIR/conf directory
   eg. fwmgmtsrv:
   C:\WINDOWS\FW1\R65\FW1\conf
   logexport.ini:
   [Fields_Info]
included_fields = num,date,time,orig,origin_id,type,action,alert,i/f_name,
i/f_dir,product,rule,src,dst,proto,service,s_port,xlatesrc,xlatedst,
nat_rulenum,nat_addtnl_rulenum,xlatesport,xlatedport,user,
partner,community,session_id,ipv6_src,ipv6_dst,
srckeyid,dstkeyid,CookieI,CookieR,msgid,elapsed,
bytes,packets,start_time,snid,ua_snid,d_name,id_src,ua_operation,
sso_type_desc,app_name,auth_domain,uname4domain,wa_headers,
result_desc,r_dest,comment,url,redirect_url,enc_desc,e2e_enc_desc,
auth_result,attack,log_sys_message,
rule_uid,rule_name,service_id,resource,reason,cat_server,
dstname,SOAP Method,category,ICMP,message_info,
TCP flags,rpc_prog,Total logs,
Suppressed logs,DCE-RPC Interface UUID,Packet info,
message,ip_id,ip_len,ip_offset,fragments_dropped,during_sec
3. Use OPSEC LEA tools to extract event log records instead of export via fwm logexport

Once the ASCII log files are available for processing, my fw1logsearch.pl script can be used to find complex patterns of interest.  Any matching records found by fw1logsearch will be output with an initial FW1 header line so that fw1logsearch can be used iteratively, to build very complex search criteria.  fw1logsearch can also write out a discard file allowing completely negative logic searches resulting in 100% of the input data separated into a match file and a didn’t match file.  Some examples of how I’ve used it are shown here:

gunzip -c fwlogs/2009*gz | \
fw1logsearch.pl --allinclude \
-S '10\.1\.1[1359]\.|10\.2\.1[01]\.|192\.168\.2[245]\.' \
-d '10\.1\.1[1359]\.|10\.2\.1[01]\.|192\.168\.2[245]\.' \
-p '^1310$|^1411$|^1812$|^455' | \
fw1logsearch.pl -S '192\.168\.22\.14$|10\.2\.11\.12$' |\
fw1logsearch.pl --allexclude \
-S '^192\.168\.24\.12$' -P '^1310$' --rejectfile 192-168-24-12-port-1310.txt

Line by line:
1. Unzip the compressed ASCII log files, feed them to the first instance of fw1logsearch.pl
2. First fw1logsearch – all conditions must be true for any events to match
Source address must NOT be in any of the following regex ranges:
10.1.11.* 10.1.13.* 10.1.15.* 10.1.19.*
10.2.10.* 10.2.11.*
192.168.22.* 192.168.24.* 192.168.25.*
Destination address must be in one of the same following regex ranges.
Service (destination port) must be one of:
Exactly port: 1310, 1411, 1812, or any port starting with 455
No protocol is specified, so it will match either TCP or UDP

fw1logsearch.pl will output any matching events to stdout, including a FW1 log header line, so the next instance of fw1logsearch.pl continues filtering the result set.

3. The second fw1logsearch.pl specifies Source Address must not be any of the following
192.168.22.14

10.2.11.12

4. The last fw1logsearch.pl excludes port 1310 from 192.168.24.12, and puts all those records into a separate reject file, while writing the other records to stdout.

This script has been used to process over 4 billion records within the project I wrote it for – and precisely found all the use of particular business cases I needed to modify.  The result was zero outages and no unintended business interruption.

Basic syntax/help file:

Usage:  fw1logsearch.pl
[-a|–incaction|-A|–excaction <action regex>]
[-p|–incservice|-P|–excservice <dst port regex>]
[-b|–incs_port|-B|–excs_port <src port regex>]
[-s|–incsrc|-S|–excsrc <src regex>]
[-d|–incdst|-D|–excdst <dst regex>]
[-o|–incorig|-O|–excorig <fw regex>]
[-r|–incrule|-R|–excrule <rule-number regex>]
[-t|–incproto|-T|–excproto <proto regex>]

[–dnscache <dns-cache-file>]
[–resolveip]
[–allinclude]
[–allexclude]
[–rejectfile <file>]
[–debug <level>]

fw1logsearch.pl will search a fwm logexport text file for regex patterns specified for supported columns (such as service, src, dst, rule, action, proto and orig).

Include and exclude regex matches may be specified on the same line, although they both will include (print) a line or exclude (reject) a line based on single matches.  Allinclude or Allexclude must be specified to force a match
only on all specified column regex patterns.

Regex patterns can be enclosed with single quotes to include characters that are special to the shell, such as the ‘or’ (|) operator.

Header will be output only if there are any matching lines.

Example invocations:
$ cat 2008-07-07*txt | \
fw1logsearch.pl \
-p ’53|domain’ \
-d ‘192.168.1.2|host1|10.10.1.2|host2’ \
-o ‘192.168.2.3|10.10.2.4|10.10.4.5’ \
-S ‘64.65.66.67|32.33.34.35|10.10.*|192.168.*’ \
–resolveip
Will require destination port (service) to be 53, destination IP to be any of 192.168.1.2, host1, 10.10.1.2, or host2  the reporting firewall (origin) to be any of 192.168.2.3, 10.10.2.4, or 10.10.4.5  and the source IP must not be
any of 64.65.66.67, 32.33.34.35, 10.10.*, or 192.168.*  Any lines that match this criteria, will display and the orig, src, and dst columns will use the default DNS cache file (dynamically built/managed) to perform name resolution, replacing the IP addresses where possible.

Include regex patterns:
-a  –incaction    Rule action (accept, deny)
-b  –incs_port    Source port (s_port)
-p  –incservice   Destination port (service)
-s  –incsrc       Source IP|hostname
-d  –incdst       Destination IP|hostname
-o  –incorig      Reporting FW IP|hostname
-r  –incrule      Rule number that triggered entry
-t  –incproto     Protocol of connection

Exclude regex patterns:
-A  –excaction    Rule action (accept, deny)
-B  –excs_port    Source port (s_port)
-P  –excservice   Destination port (service)
-S  –excsrc       Source IP|hostname
-D  –excdst       Destination IP|hostname
-O  –excorig      Reporting FW IP|hostname
-R  –excrule      Rule number that triggered entry
-T  –excproto     Protocol of connection

Other options:
–debug {level} Turn on debugging
–dnscache      Specify location of DNS cache file to be used with
the Resolve IPs option
–resolveip     Resolve IPs for orig, src, and dst columns AFTER filtering
–rejectfile    Write out all rejected lines to a specified file

Download fw1logsearch.pl