Installing daemonlogger on OpenWRT natively, LIKE A BOSS

It's that easy... and it's going in the next stable release (Attitude Adjustment, now in beta) :-D

1. Update the package listing from main repository...

root@OpenWrt:~# opkg update
Downloading http://downloads.openwrt.org/attitude_adjustment/12.09-beta/ar71xx/nand/packages//Packages.gz.
Inflating http://downloads.openwrt.org/attitude_adjustment/12.09-beta/ar71xx/nand/packages//Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker.

2. Install the daemonlogger package...

root@OpenWrt:~# opkg install daemonlogger
Installing daemonlogger (1.2.1-1) to root...
Downloading http://downloads.openwrt.org/attitude_adjustment/12.09-beta/ar71xx/nand/packages//daemonlogger_1.2.1-1_ar71xx.ipk.
Installing libpcap (1.1.1-2) to root...
Downloading http://downloads.openwrt.org/attitude_adjustment/12.09-beta/ar71xx/nand/packages//libpcap_1.1.1-2_ar71xx.ipk.
Installing libdnet (1.11-2) to root...
Downloading http://downloads.openwrt.org/attitude_adjustment/12.09-beta/ar71xx/nand/packages//libdnet_1.11-2_ar71xx.ipk.
Configuring libdnet.
Configuring libpcap.
Configuring daemonlogger.

3. Profit!

root@OpenWrt:~# daemonlogger -v
[-] Log filename set to "daemonlogger.pcap"
[-] Pidfile configured to "daemonlogger.pid"
[-] Pidpath configured to "/var/run"
[-] Rollover size set to 2147483648 bytes
[-] Rollover time configured for 0 seconds
[-] Pruning behavior set to oldest IN DIRECTORY

-*> DaemonLogger <*-
Version 1.2.1
By Martin Roesch
(C) Copyright 2006-2007 Sourcefire Inc., All rights reserved

OpenWRT on a MikroTik RouterBOARD 450G

w00t.

vineyard@token:~/openwrt/trunk/bin/ar71xx$ ssh root@192.168.1.1
root@192.168.1.1's password: 

BusyBox v1.19.4 (2012-10-14 20:32:51 EDT) built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 BARRIER BREAKER (Bleeding Edge, r33773)
 -----------------------------------------------------
  * 1/2 oz Galliano         Pour all ingredients into
  * 4 oz cold Coffee        an irish coffee mug filled
  * 1 1/2 oz Dark Rum       with crushed ice. Stir.
  * 2 tsp. Creme de Cacao
 -----------------------------------------------------

root@OpenWrt:~# uname -a
Linux OpenWrt 3.3.8 #2 Sun Oct 14 21:51:05 EDT 2012 mips GNU/Linux

root@OpenWrt:~# cat /proc/cpuinfo 
system type  : Atheros AR7161 rev 2
machine   : MikroTik RouterBOARD 450G
processor  : 0
cpu model  : MIPS 24Kc V7.4
BogoMIPS  : 452.19
wait instruction : yes
microsecond timers : yes
tlb_entries  : 16
extra interrupt vector : yes
hardware watchpoint : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
ASEs implemented : mips16
shadow register sets : 1
kscratch registers : 0
core   : 0
VCED exceptions  : not available
VCEI exceptions  : not available

Symmetric RSS (Receive-Side Scaling)

New direction. This is just damn cool. These folks have written some custom drivers that exploit collisions in the original RSS (Receive-Side Scaling) load-balancing algorithm developed by Microsoft,  such that the RX queues on the NIC end up getting properly 5-tuple load-balanced. This allows a monitoring tool to leverage the locality of reference and cache coherency inherent with having both sides of a given connection being steered to discrete CPU cores for analysis. By manipulating the secret key used in the cryptographic hash function employed by RSS, these researchers appear to have achieved IDS-optimized load-balancing completely in hardware using commodity network cards:

http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf

I need to see how this compares with similar work by Luca Deri and friends:

http://www.ntop.org/pf_ring/hardware-based-symmetric-flow-balancing-in-dna/

On the plus side, this technique appears to have already made it into the code for the DNA drivers, and a patch has recently been committed to enable this functionality for libpcap-based applications:

http://listgateway.unipi.it/pipermail/ntop-misc/2012-July/003037.html

Unfortunately, at present it seems that the DNA drivers can only be used by one network monitoring application at a time. None of this inherently solves my virtualization problem, but it's a big step in the right direction.

Stay tuned...

It's alive...

82599-based 10Gb NIC direct-mapped via PCIe SR-IOV into a KVM-paravirtualized Ubuntu 12.04. Initial test run looks very promising. Full-speed packet capture with zero copy and zero packet loss, thanks to the PF_RING DNA drivers running *inside the virtual machine*

root@randy:~# dmidecode | grep Vendor
 Vendor: Bochs

root@randy:~# dmesg | grep KVM | sed 's/\[[^]]*\]//'
 Booting paravirtualized kernel on KVM
 KVM setup async PF for cpu 0

root@randy:~# dmesg | grep ixgbe | grep dna0 | sed 's/\[[^]]*\]//'
 ixgbe 0000:00:06.0: dna0: MAC: 2, PHY: 2, PBA No: 400900-000
 ixgbe 0000:00:06.0: dna0: Enabled Features: RxQ: 16 TxQ: 16 FdirHash
 ixgbe 0000:00:06.0: dna0: Intel(R) 10 Gigabit Network Connection
 ixgbe 0000:00:06.0: dna0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

root@randy:~# dmesg | grep PF_RING | sed 's/\[[^]]*\]//'
 [PF_RING] Welcome to PF_RING 5.4.5 ($Revision: 5614$)
 [PF_RING] registered /proc/net/pf_ring/
 [PF_RING] Min # ring slots 4096
 [PF_RING] Slot version     14
 [PF_RING] Capture TX       Yes [RX+TX]
 [PF_RING] Transparent Mode 0
 [PF_RING] IP Defragment    No
 [PF_RING] Initialized correctly

root@randy:~# tcpdump -i dna0 -s0 -w /dev/null
tcpdump: listening on dna0, link-type EN10MB (Ethernet), capture size 8192 bytes
750380 packets captured
750380 packets received by filter
0 packets dropped by kernel

w00t! Show me the 10Gbps PF_RING love!

Many thanks to Luca Deri and Silicom USA for graciously supplying me with a shiny new dual-port Intel® X520-based 10 Gigabit PE210G2i9-T accelerated packet-capture card.

Now I have all of the proper network security implements of destruction to support wire-speed zero-copy DMA PF_RING deep-packet inspection at 10Gbps:

root@token:~# lspci | cut -d':' -f3 | grep Gigabit
 Intel Corporation 82579LM Gigabit Network Connection (rev 05)
 Intel Corporation I350 Gigabit Network Connection (rev 01)
 Intel Corporation I350 Gigabit Network Connection (rev 01)
 Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
 Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)

root@token:~# dmesg | cut -d']' -f2 | grep NIC | grep dna
 igb: dna0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
 ixgbe 0000:05:00.0: dna1: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Watch this space for more Ninja Master Packet Assassin™ goodness...

Security Garlic: A Preview of Coming Attractions

Upon discovering the wonders of PF_RING with DNA, OpenvSwitch L4-hashed port-mirroring, Intel® VT-d / SR-IOV, KVM with VirtIO, and Security Onion, I just knew that these incredibly powerful tools belonged together.

So I'd say this a very early Alpha-stage implementation of what I'm looking to accomplish and document, but I have high hopes for success. The goal is to use the PF_RING DNA drivers (and possibly vPF_RING) to perform transport-layer 5-tuple hashing and load-balancing in hardware, and then hand off the resulting pieces to a team of virtualized network inspection engines for deeper analysis. On the right hardware (modern higher-end Intel NICs), DNA and SR-IOV allow us to perform all this work using Zero-Copy Direct Memory Access, eliminating the need to burn CPU cycles simply to handle interrupts from your NIC. That means more horsepower to inspect all those packets.

UPDATE: I've discovered that OpenvSwitch's L4-hashing mechanism for load-balancing currently only works when LACP is enabled on a bonded link. LACP breaks the technique I'm employing here because it requires that all of the composite members of the port-channel be connected to the same endpoint device. I may have to write some patches for OpenvSwitch to get this working the way I'd like... in the meantime this is still doable with PF_RING.

UPDATE 2: I may have spoken too soon. Perhaps this has already been addressed: http://openvswitch.org/pipermail/dev/2011-May/008934.html

These Are a Few of My Favorite Things...

Rat's nests of CAT5, tracing with ping, these are a few of my favorite things... Especially when everything's all neatly connected to a single device that magically does everything I need. I'm still going to publish Part III of my guide on building Kenny, but in the meantime, I've found a fantastic replacement for a whole pile of old Linksys gear. I haven't named him yet, but I'm learning to love my new Mikrotik RouterBoard RB2011L, which I acquired for a quite reasonable $99 from r0c-n0c.com.

The user interface for this RouterOS-based device is awkward, non-intuitive, and very poorly documented. That said, this is an incredibly powerful device. I recommend that everyone buy one as soon as possible, before they stop making them. There are OpenWRT works-in-progress for other Mikrotik RouterBoard platforms, and I'm crossing my fingers that this one will be appearing on that list Real Soon Now.

Network Monitoring at Home for Fun and Profit (Part II)

This is pretty complicated stuff, so I'm going to start out with a fairly long-winded and familiar analogy. If you're already well-versed on terms like MAC addresses and ARP caches, and the challenges of port mirroring, feel free to skip this part. Or not. You might learn something. I sure did. :-)

To expand on a time-honored metaphor often taught in computer networking classes, imagine that your home network is the postal service. It's the postal service's job to sort and deliver mail to the appropriate recipient, presumably at their home address. The postal service employs mail carriers to physically shuffle millions of properly addressed, stamped, and postmarked envelopes around the country to wherever they need to go each day.

ZFS and DTrace on Ubuntu Linux

I know what I'll be doing this weekend...

"So, I had heard about ZFS and DTrace running on Linux. People have been working on them for years. Basically, since the time the technologies came out on Solaris. So, I decided to follow up on these ports and see what could be done. Luckily, I had great success in getting both to run on an Ubuntu server of mine. If you would like to do so as well, just follow the steps below."

http://liberumvir.com/2012/06/01/zfs-and-dtrace-running-on-ubuntu.html

Daemonlogger native package now in OpenWRT trunk!

Success! My patch for building Daemonlogger as a native OpenWRT package has been accepted into the mainline distribution and committed to trunk. Pre-built binary packages are now available for all supported architectures in the nightly snapshots tree. You'll find the appropriate .ipk for Kenny (a WRT150N) and similar Broadcom-based routers here.

Unfortunately these packages only work on the latest trunk firmware builds at the moment, and the 3.2 kernel along with the extra software included in these builds does not leave enough free JFFS space or usable RAM to run daemonlogger effectively. I'm trying to convince the developers to include this in the next stable release (Backfire 10.03.2) based on the 2.6 kernel, but no luck yet.

For the time being you can still grab my binary packages here from my GitHub repository. These *do* install and run cleanly on the current stable version (Backfire 10.03.1). For most Broadcom-based WRT-type devices, you'll want the brcm47xx build.

UPDATE: Binary daemonlogger .ipk packages for ALL OpenWRT-supported embedded devices and platforms are now available here: http://bit.ly/dl-owrt

Meet Kenny :-)

Everything on my network is named after a character from South Park, and Kenny seemed like an appropriate name for this little fella because he died many, many horrible deaths along the way to getting this right.

After making all that fuss about missing those big boxes with shiny logos, I decided that Kenny deserved his very own piece of flair. Strangely, no one seems to make OpenWRT badges, so I got him the next best thing - a brushed aluminum Tux badge...

Network Monitoring at Home for Fun and Profit (Part I)

Who says you can't have professional, enterprise-grade network monitoring capabilities over your home or small office network for less than $100 worth of commodity off-the-shelf hardware? Not me! Often times, the surest way to get something difficult accomplished is to try to convince me that it can't be done. Impossible you say? That's crazy talk.

Recently I left a position that had afforded me the opportunity to play around with all kinds of fancy, expensive toys - the kind you might see sporting flashy logos with big names like Cisco and Gigamon and NetOptics. It wasn't very long after parting ways with my beloved magical boxes of networking tricks before I started getting that itch to tinker. Bad. I wanted my lab back, dammit.

WRT-SPAN Block Diagram (rough draft)

I've got a friend helping me turn this into something a bit more legible (and preferably in SVG format so I can revise it if necessary), but here is a rough draft of what's going on inside my makeshift aggregating ethernet tap:

Hybrid HDD + SSD RAID1

Not really security-related at all, but this is just damn cool. After killing three SSDs in 18 months (thankfully the drives were under warranty, but my data wasn't), I went looking for a better solution. When SSDs crap out, they go suddenly and catastrophically. There's no putting them in the freezer or swapping controller boards with another unit like you can with a mechanical hard drive. Once the sector mapping tables - which are constantly being updated due to wear-leveling algorithms - get corrupted, you're pretty much not getting your data back. Turns out there's a simple way to combine the speed benefits of an SSD with the reliability of an HDD-backed RAID1 mirror.

The lab is growing...

So I happened to find myself with a bunch of old Linksys wireless routers lying around. I like to collect such things from friends and relatives after they've been "fried" in hopes of one day bringing them back to life and hacking them into something useful. We all know how that goes. These poor, forgotten toys spent most of the last few years gathering dust in my basement, longing for someone to play with them. Until now...