Cut Squid log data


Use the below to cut some data out of squid log that we dont need. Will cut up to the point added in the quotes (”)

From

squidlog.log.18:94.236.109.235 851 - - [28/Sep/2011:19:01:37 +0100] "GET http://someurl HTTP/1.
1" 200 36921 "-" "-" "-" TCP_MISS:SERVER

To

94.236.109.235 866 - - [28/Sep/2011:19:13:57 +0100] "GET someurl HTTP/1.1" 200 38214 "-" "-" "-" TCP_MISS

Using

cut -d ':' -f2,3,4,5,6, squidlog.txt
Advertisements

Awk and more awk..


The logs they never end…More logs and more logs and more problems with more logs…Needed to look at some apache access logs today. Handy to be able to use awk to filter the data to make it a bit more visible to whats going on. Using the below we can count the number of occurances for the specified part of the line. In the below example its getting the number of occurances for the IP’s in the access log but you can amend this to see the request headers or status codes depending on their location on the line using awk.

Example

Access Log

132.17.14.252 - - [09/Sep/2011:04:16:41 +0100] "GET /something.html HTTP/1.1" 200 7031 46785 "-" "-" blah.something.net

Awk

zcat apache.access.log.1.gz | awk {'print $1'} | sort | uniq -c | sort -nr | head -10

Result

 485487 175.12.15.200
    557 216.151.121.50
    506 80.141.40.132
    486 218.156.138.239
    475 79.142.40.105
    452 79.142.40.115
    450 79.142.41.118
    444 218.156.138.238
    441 87.134.71.123
    436 218.156.138.211

Grep for IP addresses


Another fun day at work. Today’s fun was attempting to extract some IP addresses from a wonderful log file filled with tildes!

Life made a little easier using the below grep to get out the IP format (192.168.1.1) etc. Pretty useful to know.

Example

Log with Tildes (ahhhhhhhh)

20110906000229~83.223.170.23~

Grep

egrep "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" -o