Installing daemonlogger on OpenWRT natively, LIKE A BOSS

It's that easy... and it's going in the next stable release (Attitude Adjustment, now in beta) :-D

1. Update the package listing from main repository...

root@OpenWrt:~# opkg update
Downloading http://downloads.openwrt.org/attitude_adjustment/12.09-beta/ar71xx/nand/packages//Packages.gz.
Inflating http://downloads.openwrt.org/attitude_adjustment/12.09-beta/ar71xx/nand/packages//Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker.

2. Install the daemonlogger package...

root@OpenWrt:~# opkg install daemonlogger
Installing daemonlogger (1.2.1-1) to root...
Downloading http://downloads.openwrt.org/attitude_adjustment/12.09-beta/ar71xx/nand/packages//daemonlogger_1.2.1-1_ar71xx.ipk.
Installing libpcap (1.1.1-2) to root...
Downloading http://downloads.openwrt.org/attitude_adjustment/12.09-beta/ar71xx/nand/packages//libpcap_1.1.1-2_ar71xx.ipk.
Installing libdnet (1.11-2) to root...
Downloading http://downloads.openwrt.org/attitude_adjustment/12.09-beta/ar71xx/nand/packages//libdnet_1.11-2_ar71xx.ipk.
Configuring libdnet.
Configuring libpcap.
Configuring daemonlogger.

3. Profit!

root@OpenWrt:~# daemonlogger -v
[-] Log filename set to "daemonlogger.pcap"
[-] Pidfile configured to "daemonlogger.pid"
[-] Pidpath configured to "/var/run"
[-] Rollover size set to 2147483648 bytes
[-] Rollover time configured for 0 seconds
[-] Pruning behavior set to oldest IN DIRECTORY

-*> DaemonLogger <*-
Version 1.2.1
By Martin Roesch
(C) Copyright 2006-2007 Sourcefire Inc., All rights reserved

OpenWRT on a MikroTik RouterBOARD 450G

w00t.

vineyard@token:~/openwrt/trunk/bin/ar71xx$ ssh root@192.168.1.1
root@192.168.1.1's password: 

BusyBox v1.19.4 (2012-10-14 20:32:51 EDT) built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 BARRIER BREAKER (Bleeding Edge, r33773)
 -----------------------------------------------------
  * 1/2 oz Galliano         Pour all ingredients into
  * 4 oz cold Coffee        an irish coffee mug filled
  * 1 1/2 oz Dark Rum       with crushed ice. Stir.
  * 2 tsp. Creme de Cacao
 -----------------------------------------------------

root@OpenWrt:~# uname -a
Linux OpenWrt 3.3.8 #2 Sun Oct 14 21:51:05 EDT 2012 mips GNU/Linux

root@OpenWrt:~# cat /proc/cpuinfo 
system type  : Atheros AR7161 rev 2
machine   : MikroTik RouterBOARD 450G
processor  : 0
cpu model  : MIPS 24Kc V7.4
BogoMIPS  : 452.19
wait instruction : yes
microsecond timers : yes
tlb_entries  : 16
extra interrupt vector : yes
hardware watchpoint : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
ASEs implemented : mips16
shadow register sets : 1
kscratch registers : 0
core   : 0
VCED exceptions  : not available
VCEI exceptions  : not available

Symmetric RSS (Receive-Side Scaling)

New direction. This is just damn cool. These folks have written some custom drivers that exploit collisions in the original RSS (Receive-Side Scaling) load-balancing algorithm developed by Microsoft,  such that the RX queues on the NIC end up getting properly 5-tuple load-balanced. This allows a monitoring tool to leverage the locality of reference and cache coherency inherent with having both sides of a given connection being steered to discrete CPU cores for analysis. By manipulating the secret key used in the cryptographic hash function employed by RSS, these researchers appear to have achieved IDS-optimized load-balancing completely in hardware using commodity network cards:

http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf

I need to see how this compares with similar work by Luca Deri and friends:

http://www.ntop.org/pf_ring/hardware-based-symmetric-flow-balancing-in-dna/

On the plus side, this technique appears to have already made it into the code for the DNA drivers, and a patch has recently been committed to enable this functionality for libpcap-based applications:

http://listgateway.unipi.it/pipermail/ntop-misc/2012-July/003037.html

Unfortunately, at present it seems that the DNA drivers can only be used by one network monitoring application at a time. None of this inherently solves my virtualization problem, but it's a big step in the right direction.

Stay tuned...

It's alive...

82599-based 10Gb NIC direct-mapped via PCIe SR-IOV into a KVM-paravirtualized Ubuntu 12.04. Initial test run looks very promising. Full-speed packet capture with zero copy and zero packet loss, thanks to the PF_RING DNA drivers running *inside the virtual machine*

root@randy:~# dmidecode | grep Vendor
 Vendor: Bochs

root@randy:~# dmesg | grep KVM | sed 's/\[[^]]*\]//'
 Booting paravirtualized kernel on KVM
 KVM setup async PF for cpu 0

root@randy:~# dmesg | grep ixgbe | grep dna0 | sed 's/\[[^]]*\]//'
 ixgbe 0000:00:06.0: dna0: MAC: 2, PHY: 2, PBA No: 400900-000
 ixgbe 0000:00:06.0: dna0: Enabled Features: RxQ: 16 TxQ: 16 FdirHash
 ixgbe 0000:00:06.0: dna0: Intel(R) 10 Gigabit Network Connection
 ixgbe 0000:00:06.0: dna0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

root@randy:~# dmesg | grep PF_RING | sed 's/\[[^]]*\]//'
 [PF_RING] Welcome to PF_RING 5.4.5 ($Revision: 5614$)
 [PF_RING] registered /proc/net/pf_ring/
 [PF_RING] Min # ring slots 4096
 [PF_RING] Slot version     14
 [PF_RING] Capture TX       Yes [RX+TX]
 [PF_RING] Transparent Mode 0
 [PF_RING] IP Defragment    No
 [PF_RING] Initialized correctly

root@randy:~# tcpdump -i dna0 -s0 -w /dev/null
tcpdump: listening on dna0, link-type EN10MB (Ethernet), capture size 8192 bytes
750380 packets captured
750380 packets received by filter
0 packets dropped by kernel

w00t! Show me the 10Gbps PF_RING love!

Many thanks to Luca Deri and Silicom USA for graciously supplying me with a shiny new dual-port Intel® X520-based 10 Gigabit PE210G2i9-T accelerated packet-capture card.

Now I have all of the proper network security implements of destruction to support wire-speed zero-copy DMA PF_RING deep-packet inspection at 10Gbps:

root@token:~# lspci | cut -d':' -f3 | grep Gigabit
 Intel Corporation 82579LM Gigabit Network Connection (rev 05)
 Intel Corporation I350 Gigabit Network Connection (rev 01)
 Intel Corporation I350 Gigabit Network Connection (rev 01)
 Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
 Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)

root@token:~# dmesg | cut -d']' -f2 | grep NIC | grep dna
 igb: dna0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
 ixgbe 0000:05:00.0: dna1: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Watch this space for more Ninja Master Packet Assassin™ goodness...

Security Garlic: A Preview of Coming Attractions

Upon discovering the wonders of PF_RING with DNA, OpenvSwitch L4-hashed port-mirroring, Intel® VT-d / SR-IOV, KVM with VirtIO, and Security Onion, I just knew that these incredibly powerful tools belonged together.

So I'd say this a very early Alpha-stage implementation of what I'm looking to accomplish and document, but I have high hopes for success. The goal is to use the PF_RING DNA drivers (and possibly vPF_RING) to perform transport-layer 5-tuple hashing and load-balancing in hardware, and then hand off the resulting pieces to a team of virtualized network inspection engines for deeper analysis. On the right hardware (modern higher-end Intel NICs), DNA and SR-IOV allow us to perform all this work using Zero-Copy Direct Memory Access, eliminating the need to burn CPU cycles simply to handle interrupts from your NIC. That means more horsepower to inspect all those packets.

UPDATE: I've discovered that OpenvSwitch's L4-hashing mechanism for load-balancing currently only works when LACP is enabled on a bonded link. LACP breaks the technique I'm employing here because it requires that all of the composite members of the port-channel be connected to the same endpoint device. I may have to write some patches for OpenvSwitch to get this working the way I'd like... in the meantime this is still doable with PF_RING.

UPDATE 2: I may have spoken too soon. Perhaps this has already been addressed: http://openvswitch.org/pipermail/dev/2011-May/008934.html

These Are a Few of My Favorite Things...

Rat's nests of CAT5, tracing with ping, these are a few of my favorite things... Especially when everything's all neatly connected to a single device that magically does everything I need. I'm still going to publish Part III of my guide on building Kenny, but in the meantime, I've found a fantastic replacement for a whole pile of old Linksys gear. I haven't named him yet, but I'm learning to love my new Mikrotik RouterBoard RB2011L, which I acquired for a quite reasonable $99 from r0c-n0c.com.

The user interface for this RouterOS-based device is awkward, non-intuitive, and very poorly documented. That said, this is an incredibly powerful device. I recommend that everyone buy one as soon as possible, before they stop making them. There are OpenWRT works-in-progress for other Mikrotik RouterBoard platforms, and I'm crossing my fingers that this one will be appearing on that list Real Soon Now.

Network Monitoring at Home for Fun and Profit (Part II)

This is pretty complicated stuff, so I'm going to start out with a fairly long-winded and familiar analogy. If you're already well-versed on terms like MAC addresses and ARP caches, and the challenges of port mirroring, feel free to skip this part. Or not. You might learn something. I sure did. :-)

To expand on a time-honored metaphor often taught in computer networking classes, imagine that your home network is the postal service. It's the postal service's job to sort and deliver mail to the appropriate recipient, presumably at their home address. The postal service employs mail carriers to physically shuffle millions of properly addressed, stamped, and postmarked envelopes around the country to wherever they need to go each day.