Last week I wrote a quick little tutorial so that one could get started using tshark. In this post I want to look at different ways of viewing the same data using a tool called zeek. Zeek is often referred to as a packet examination ‘framework’ as it allows you to see what is happening, the whos, wheres and whats within the traffic. Zeek is often deployed along side other tools like snort, suricata and/or moloch.
Since we will be examining pcaps, not live traffic we will again be going with the ‘-r’ option as we did with previous posts covering tcpdump and tshark.
$ ls
ctf-dump-v2.pcapng ctf.pcap zeek.script
$ zeek -Cr ctf.pcap
$ ls
conn.log dns.log ftp.log ntp.log smtp.log ssl.log zeek.script
ctf-dump-v2.pcapng dpd.log http.log packet_filter.log snmp.log weird.log
ctf.pcap files.log mysql.log sip.log ssh.log x509.logYou can see, after we read in our pcap with zeek a bunch of *.log files were created. You can guess what kind of information is in each log based on it’s name. To view logs nativly, zeek has a tool called ‘zeek-cut’ that allows you to format and view what you’d like. If you use just zeek-cut you will get the default columns:
$ head dns.log | zeek-cut
1613159462.737544 Ci2kw63INthRjNjuae 157.230.15.223 57199 67.207.67.3 53 udp 6601 - 223.15.230.157.in-addr.arpa 1C_INTERNET 12 PTR 3 NXDOMAIN F F T F 0 - - FWhat are these columns you ask?! Good question. We can see what are all our options are as far as data within this log by simply looking at the very beginning of the file:
$ head dns.log
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path dns
#open 2021-04-16-17-46-03
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id rtt query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected
#types time string addr port addr port enum count interval string count string count string count string bool bool bool bool count vector[string] vector[interval] boolFields we can extract/view from this log are listed after the #fields above.
An aside: A bit about source/destination vs originator/responder. In zeek the one who initiates a request, whether by a syn or what have you, is the originator and the one responding, ie, a syn-ack is the responder. They do not use the lexicon of source and destination. Which, I think, is kind of cool as one of the things you do with tcpdump a lot is filter by syns or syn-acks and here that work is already done for you.
Back to parsing this log file. Using zeek-cut, let’s pull out the id.orig_h, resp_p and the query. I only pipe it to head for brevity.
$ cat dns.log | zeek-cut id.orig_h id.resp_p query | sort | uniq | head
10.10.10.101 53 assets.msn.com
10.10.10.101 53 cdn.content.prod.cms.msn.com
10.10.10.101 53 debug.opendns.com
10.10.10.101 53 portal.mango.local
10.10.10.101 53 sw-ec.mango.local
10.10.10.101 53 sync.hydra.opendns.com
10.10.10.101 53 www.gstatic.com
10.10.10.101 53 www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com
127.0.0.1 53 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.d.0.0.0.0.4.0.0.8.8.a.4.0.6.2.ip6.arpa
127.0.0.1 53 1.0.0.0.5.7.e.1.0.0.0.0.0.0.0.0.0.d.0.0.0.0.4.0.0.8.8.a.4.0.6.2.ip6.arpaThis information is exactly the same information we pulled out of the file last week with tshark. Zeek is an awesome tool because the logs, once extracted from live capture or a pcap can be held onto for a long time because in relation to the hard-drive space needed for a pcap, Zeek logs take up very little space. You can refer to these artifacts later and retain for much longer/easier than trying to retain pcaps.
Another pro for zeek is that parsing through a log file is computationally super fast when compared to tshark or even tcpdump trying to look through an entire pcap every time you do a filter. So getting information out of your data, once read through zeek is FAST!
So to briefly recap, to get started with zeek-cut looking at your logs, head a log you are interested in, see the possible columns and then use zeek-cut to parse out what you are interested in. Another thing I demonstrated last week in my tshark post was pulling out all the usernames used to login with mysql. Can we quickly do the thing with zeek?
$ ls *.log
conn.log dpd.log ftp.log mysql.log packet_filter.log smtp.log ssh.log weird.log
dns.log files.log http.log ntp.log sip.log snmp.log ssl.log x509.logWe see we have a mysql.log and the next step is to head it and see the columns.
$ head mysql.log
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path mysql
#open 2021-04-16-17-46-03
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p cmd arg success rows response
#types time string addr port addr port string string bool count stringThe three columns that stand out as possibilities that could help us reach our goal of getting all the username’s/passwords to log in would be cmd, arg, success, rows and response. One of the cmd is ‘login’ so if we grep for login and show associated arg we are able to see all the usernames:
$ cat mysql.log | zeek-cut cmd arg | grep login | sort | uniq -c
2 login 8TmveSod
12 login admin
4 login admin@example.com
1 login flag
4 login jamfsoftware
12 login mysql
140 login root
4 login superdba
12 login test
12 login user
4 login username
2 login wdxhpxxKTo briefly look back, here was us last week doing the same thing with tshark:
$ tshark -r ctf.pcap -Y 'mysql' -T fields -e mysql.user | sort | uniq -c
963
2 8TmveSod
12 admin
4 admin@example.com
1 flag
4 jamfsoftware
12 mysql
140 root
4 superdba
12 test
12 user
4 username
2 wdxhpxxKOne more really cool thing to mention about Zeek before we shift over into looking at the same data in JSON format using jq is that of the uid. Let’s say for whatever reason, you are super interested in someone logging in with the username flag. In zeek, every single log has a UID, which is a unique identifier of traffic consisting of the same 5-tuple or source IP address/port number, destination IP address/port number and the protocol in use. So if we include the UID in the login associated with flag we could then grep all of our logs for that UID to see all the associated traffic.
$ cat mysql.log | zeek-cut cmd arg uid | grep flag
login flag C4nJ2N3ksR7OfGiU9k
$ grep C4nJ2N3ksR7OfGiU9k *.log
conn.log:1613168140.809131 C4nJ2N3ksR7OfGiU9k 157.230.15.223 45330 172.17.0.2 3306 tcp - 0.011629 443 1438 SF - - 0 ShAdtDTaFf 48 3446 38 4868 -
dpd.log:1613168140.809956 C4nJ2N3ksR7OfGiU9k 157.230.15.223 45330 172.17.0.2 3306 tcp MYSQL Binpac exception: binpac exception: out_of_bound: LengthEncodedIntegerLookahead:i4: 8 > 6
mysql.log:1613168140.809676 C4nJ2N3ksR7OfGiU9k 157.230.15.223 45330 172.17.0.2 3306 login flag - - -
mysql.log:1613168140.809750 C4nJ2N3ksR7OfGiU9k 157.230.15.223 45330 172.17.0.2 3306 unknown-167 \xb3\x12\xd815'\x07%\x814\xfeP\x9b\x1a\xfd\xae\xc85\xee - - -
mysql.log:1613168140.809838 C4nJ2N3ksR7OfGiU9k 157.230.15.223 45330 172.17.0.2 3306 query \x00\x01select @@version_comment limit 1-- -We have easily located associated traffic with the mysql traffic with the login name of ‘flag’ very quickly.
Another very quick aside. A tool that’s like uid, but even more useful is called community-id. This is the same sort of idea as uid except you can take this ‘community-id’ and pivot to entirely different tools. Say we found something with traffic in zeek that was super interesting but wanted to look at the pcap. If we were using community-id we could copy it from our zeek log like we did with uid but this time search for this community-id within a tool like moloch (view flows and download pcap) and get greater context/viability.
Alright. So many quick asides today. Back to the lesson at hand. Zeek data can also be output in JSON format as opposed to simple text logs as outlined above. This is how zeek is configured at my work and is done so it can be easily ingested into our SIEM. Today we are just going to read in the same pcap and play around a bit with a tool called jq to parse our logs. Here is how we switch to a JSON format:
$ zeek -Cr ctf.pcap -e 'redef LogAscii::use_json=T;'If we head our dns.log, like we did above to search for quries our data will look much different. So much so that zeek-cut no longer works with this format 🙂
$ head dns.log
{"ts":1613159462.737544,"uid":"CyZQzA1XgYbK1dLIah","id.orig_h":"157.230.15.223","id.orig_p":57199,"id.resp_h":"67.207.67.3","id.resp_p":53,"proto":"udp","trans_id":6601,"query":"223.15.230.157.in-addr.arpa","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","rcode":3,"rcode_name":"NXDOMAIN","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1613159462.737492,"uid":"C1n5WP2f5tNp0iBXa2","id.orig_h":"157.230.15.223","id.orig_p":56994,"id.resp_h":"67.207.67.2","id.resp_p":53,"proto":"udp","trans_id":505,"query":"223.15.230.157.in-addr.arpa","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","rcode":3,"rcode_name":"NXDOMAIN","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}We now have a whole bunch of key:value pairs. Which means our log files will be slightly bigger than the plain txt ones but otherwise all the pros mentioned above still hold true here. Instead of piping to zeek-cut we are going to use jq to parse our data. To look at the first log, we will use the -s ‘.[0]’ option (which simply picks out the first thing in the index, ie the first log):
$ cat dns.log | jq -s '.[0]'
{
"ts": 1613159462.737544,
"uid": "CEDtgA2onmkOdbRSp",
"id.orig_h": "157.230.15.223",
"id.orig_p": 57199,
"id.resp_h": "67.207.67.3",
"id.resp_p": 53,
"proto": "udp",
"trans_id": 6601,
"query": "223.15.230.157.in-addr.arpa",
"qclass": 1,
"qclass_name": "C_INTERNET",
"qtype": 12,
"qtype_name": "PTR",
"rcode": 3,
"rcode_name": "NXDOMAIN",
"AA": false,
"TC": false,
"RD": true,
"RA": false,
"Z": 0,
"rejected": false
}I always find myself heading a log or looking at the first log before I really dive in. This is because I never remember what the key value is or the specific name of the interesting thing I’m looking for. This gives me a chance to look at an entire log and make out what each thing is referencing and I can make a better guess on what search term to use or how it should be formatted. Doing this first saves you a bit of time later in my opinion.
Every key, if you can remember back to the beginning of this post will correspond to a column header when we were using zeek-cut. With zeek-cut we used id.orig_h, id.resp_p and query. To do this we will use the -j (join option) with jq which will put the following things we select on the same line. We have to put ‘id.orig_h’ and ‘id.resp_p’ in brackets because their key value begins with a ‘.’ already and in order for jq to read them the syntax with the square brackets is needed. Since query doesn’t begin with a ‘.’ no brackets needed. “\n” simply means new line. Below we have a csv formatted version of what we did with zeek-cut above.
$ cat dns.log | jq -j '.["id.orig_h"], ", ", .["id.resp_p"], ", ", .query, "\n"' | sort | uniq |head
10.10.10.101, 53, assets.msn.com
10.10.10.101, 53, cdn.content.prod.cms.msn.com
10.10.10.101, 53, debug.opendns.com
10.10.10.101, 53, portal.mango.local
10.10.10.101, 53, sw-ec.mango.local
10.10.10.101, 53, sync.hydra.opendns.com
10.10.10.101, 53, www.gstatic.com
10.10.10.101, 53, www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com
127.0.0.1, 53, 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.d.0.0.0.0.4.0.0.8.8.a.4.0.6.2.ip6.arpa
127.0.0.1, 53, 1.0.0.0.5.7.e.1.0.0.0.0.0.0.0.0.0.d.0.0.0.0.4.0.0.8.8.a.4.0.6.2.ip6.arpaIf you forgot what we did with zeek-cut above i’ll spare you the work of having to scroll up:
$ cat dns.log | zeek-cut id.orig_h id.resp_p query | sort | uniq | head
10.10.10.101 53 assets.msn.com
10.10.10.101 53 cdn.content.prod.cms.msn.com
10.10.10.101 53 debug.opendns.com
10.10.10.101 53 portal.mango.local
10.10.10.101 53 sw-ec.mango.local
10.10.10.101 53 sync.hydra.opendns.com
10.10.10.101 53 www.gstatic.com
10.10.10.101 53 www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com
127.0.0.1 53 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.d.0.0.0.0.4.0.0.8.8.a.4.0.6.2.ip6.arpa
127.0.0.1 53 1.0.0.0.5.7.e.1.0.0.0.0.0.0.0.0.0.d.0.0.0.0.4.0.0.8.8.a.4.0.6.2.ip6.arpaIf we look at the mysql log I’m sure you can already make out how we could search for usernames used to login like we did with zeek-cut using jq:
$ cat mysql.log | jq -s '.[0]'
{
"ts": 1613164528.211387,
"uid": "CCk4OU1exd8KJARVSg",
"id.orig_h": "45.55.46.240",
"id.orig_p": 38550,
"id.resp_h": "157.230.15.223",
"id.resp_p": 3306,
"cmd": "login",
"arg": "8TmveSod"
}$ cat mysql.log | jq -j '.cmd, ", ", .arg, "\n"' | grep login | sort | uniq -c
2 login, 8TmveSod
12 login, admin
4 login, admin@example.com
1 login, flag
4 login, jamfsoftware
12 login, mysql
140 login, root
4 login, superdba
12 login, test
12 login, user
4 login, username
2 login, wdxhpxxKAbove I used grep to do the same sort of search that we did with zeek-cut. But, we don’t have to use grep as jq has some very cool functions built in that allow us to do comparison searching within the tool itself. This is where I think jq really shines. You can use ‘<‘ ‘>’ or ‘==’ to filter your search how ever you need. Here we just want to get all the ‘cmd’ that equal login.
$ cat mysql.log | jq 'select(.cmd == "login")' | jq -j '.cmd, " ", .arg, "\n"' | sort | uniq -c
2 login 8TmveSod
12 login admin
4 login admin@example.com
1 login flag
4 login jamfsoftware
12 login mysql
140 login root
4 login superdba
12 login test
12 login user
4 login username
2 login wdxhpxxKWith zeek-cut we zeroed in on the flag login and searched all our logs for the uid to find all relevant traffic with the associated tuple. We can do the same thing with jq no problem.
$ cat mysql.log | jq 'select(.cmd == "login" and .arg == "flag")' | jq -j '.uid, " ",.cmd, " ", .arg, "\n"' | sort | uniq -c
1 CmBHdR2a0DMQ9kfam login flag
$ cat *.log | jq 'select(.uid == "CmBHdR2a0DMQ9kfam")'
{
"ts": 1613168140.809131,
"uid": "CmBHdR2a0DMQ9kfam",
"id.orig_h": "157.230.15.223",
"id.orig_p": 45330,
"id.resp_h": "172.17.0.2",
"id.resp_p": 3306,
"proto": "tcp",
"duration": 0.011629104614257812,
"orig_bytes": 443,
"resp_bytes": 1438,
"conn_state": "SF",
"missed_bytes": 0,
"history": "ShAdtDTaFf",
"orig_pkts": 48,
"orig_ip_bytes": 3446,
"resp_pkts": 38,
"resp_ip_bytes": 4868
}
{
"ts": 1613168140.809956,
"uid": "CmBHdR2a0DMQ9kfam",
"id.orig_h": "157.230.15.223",
"id.orig_p": 45330,
"id.resp_h": "172.17.0.2",
"id.resp_p": 3306,
"proto": "tcp",
"analyzer": "MYSQL",
"failure_reason": "Binpac exception: binpac exception: out_of_bound: LengthEncodedIntegerLookahead:i4: 8 > 6"
}
{
"ts": 1613168140.809676,
"uid": "CmBHdR2a0DMQ9kfam",
"id.orig_h": "157.230.15.223",
"id.orig_p": 45330,
"id.resp_h": "172.17.0.2",
"id.resp_p": 3306,
"cmd": "login",
"arg": "flag"
}
{
"ts": 1613168140.80975,
"uid": "CmBHdR2a0DMQ9kfam",
"id.orig_h": "157.230.15.223",
"id.orig_p": 45330,
"id.resp_h": "172.17.0.2",
"id.resp_p": 3306,
"cmd": "unknown-167",
"arg": "\\xb3\\x12\\xd815'\\x07%\\x814\\xfeP\\x9b\\x1a\\xfd\\xae\\xc85\\xee"
}
{
"ts": 1613168140.809838,
"uid": "CmBHdR2a0DMQ9kfam",
"id.orig_h": "157.230.15.223",
"id.orig_p": 45330,
"id.resp_h": "172.17.0.2",
"id.resp_p": 3306,
"cmd": "query",
"arg": "\\x00\\x01select @@version_comment limit 1"
}I might have not shown the most ‘useful’ parsing within jq but I hope by showing you a few examples of how you can select based on the values of certain fields you can see how easy it is to zero in on what you are looking for. You can, for example, only display only logs that have a ip.orig_p less than 1000 in your conn.log with ease. Or, display on logs with a packet bigger than a certain size. The possibilities are endless and being able to use comparison operators in your search, I think, is just awesome.
Also, you can format your output based on whatever values in any order and to csv very easily if that’s a useful avenue for you. There is even more stuff you can do with jq, such as sorting. But I think we’ve went long enough 🙂
That’s all for today as I think I’ve rambled on long enough, with far to many asides. But i digress. Next time I’m thinking of trying to write my first zeek script. Till next time!
