linux: how to obtain traffic statistics

Hello everyone,

I have a linux-based bridge. I'd like to count how much traffic does every IP that passes throug generates. In other words I'd like to get a file like this:

src_IP  dst_IP  pkt_size
1.1.1.1 2.2.2.2 12304
3.3.3.3 4.4.4.4 430
....

Is there a tool to do it under linux?

I've done some search on ulog, however both "accounting daemon" projects seem to be abandoned (their webpages do not work). Capturing traffic with tcpdump and then analyzing the dump could be an option, however I can't see how can I get the packet size from the dump.

Please, let me know if there is a way to do it.

From serverfault facha

• Have a look here there may be something that will fit your needs.

  From Iain

• First of all let's start by mentioning a tool that you can watch this data real time: iftop

  Secondly I think it's important to mention that intrusive packet capturing and analysis like tcpdump is going to take it's toll on your interface and computing resources. Unless you setup a separate system that blindly watches traffic on the wire without touching it, such as snort, you are going to take a hit. Never forget this or you'll end up with a hobbled network and when the load goes up people will wonder why it sucks!

  ---

  (Edit: finish where I left off)

  Lastly, if you really need a permanent logging solution you'll need something that analysis and logs that packet data. For this there are lot of tools, but none that I have used recently. ngrep used to be a good way to do this, so did bmon but I don't know if that even exists any more. tcpdstat looks like it might be an option as does bandwidth although that might do MORE than you want.

  From Caleb

• In the IP header, you have a TOTAL LENGTH header that contains "the length of the datagram, measured in octets, including internet header and data." (RFC 791). If you want to have only the payload size of a packet, you need to do TOTAL LENGTH minus (IP Header length + (TCP|UDP) header length). (IP header length is in the IHL header, TCP header length is in the data offset header).

  With a basic tcpdump command, such as

  # tcpdump -s 1500 -Svni eth0 tcp and port 80
  

  I will display each TCP packet as follow

  11:58:52.114411 IP (tos 0x0, ttl 53,id 5745, offset 0, flags [DF], proto TCP (6), length 505)

  12.66.33.88.53247 > 88.231.98.32.80: Flags [P.], cksum 0x62fd (correct), seq 1193308573:1193309026, ack 2122411067, win 46, options [nop,nop,TS val 122841090 ecr 125780554], length 453
  

  The first line contains the IP header, including the total length (length 505). The second line contains the tcp header, including the payload length (length 453), which is 505 - 52 (52 being the length of the IP + TCP headers).

  If you want to automate this, you can set tcpdump to store the capture in a pcap file, and then parse the pcap with a script.

  However, if you want to do it fast without degrading your performances, you should take a look at libnetfilter_queue. That involves a bit of coding in C, but really not much. The idea is that you place a hook in netfilter to direct the trafic to you program. From there you can parse ip|tcp|udp headers and compute your information, put that in a RRD database if you want, and reinject the trafic in netfilter.

  facha : Great. Just what I needed. Thanks a lot.

  Julien Vehent : which part ? the tcpdump or the libnetfilter_queue ? :)

  facha : tcpdump. For my task it will be just enough to leave it running for a couple of days, grabbing traffic into a pcap file. Then I'll be able to convert it to a text file and parse it based on "length" fields. Writing a netfilter hook would be an overkill for my task.
  From Julien Vehent