Robert Mustacchi of Joyent suggested I write about a tool that he wrote that allows one to simulate network problems. The tool is ipdadm(1M), the internet packet disturber. The tool allows you to specify as a percentage: packet drops, packet corruption, and delays. This can be very useful when you want to test the effect of these on the behavior of a networked application.

The tool works on all network interfaces of a given zone. Be careful that you do not use this in a place where you can not undo your actions. For instance, on my first time trying this, I specified 20% of packets are to be corrupted in a zone that I was ssh-ed into. My ssh session dropped immediately. Fortunately, I had access to the global zone on the machine and was able to change the corruption rate back to 0%. (Note also that with access to the global zone, you can use zlogin to login to the zone where you are using ipdadm). Currently there is no way to specify a specific interface, IP address, or port number. The tool works on all interfaces being used within a zone.

Also note that the tool is full duplex. So if you request a 20 microsecond delay on packets, the delay is both for incoming and outgoing packets.

Here are some examples, run from the global zone.

# ipdadm -z 73e886b7-77a5-4081-aaf6-83382e3bdced infozone 73e886b7-77a5-4081-aaf6-83382e3bdced does not exist or has no ipd actions enabled#

Currently, no ipd actions are set for the zone. From a different machine:

$ scp words root@151.1.224.26:/var/tmp/words                                         100%  202KB 201.8KB/s   00:01    $ scp words root@151.1.224.26:/var/tmp/words                                         100%  202KB201.8KB/s   00:01

Let's set the drop rate to 10% and try again.

# ipdadm -z 73e886b7-77a5-4081-aaf6-83382e3bdced drop 10# ipdadm -z 73e886b7-77a5-4081-aaf6-83382e3bdced infoipd information for zone 73e886b7-77a5-4081-aaf6-83382e3bdced:       corrupt:        0% chance of packet corruption  drop:           10% chance of packet drop       delay:          0 microsecond delay per packet#

And from the other machine:

$ scp words root@151.1.224.26:/var/tmp/words                                         100%  202KB  22.4KB/s   00:09    $ scp words root@151.1.224.26:/var/tmp/words                                         100%  202KB  40.4KB/s   00:05$

A 10% drop rate resulted in 5 to 10 times longer. At a 30% drop rate, scp stalled aftercopying 71% of the file, and did not finish. Running netstat(1M) shows retransmissions (to be expected with packets being dropped).

# netstat -s 5 | grep tcpRetrans     tcpRetransSegs      =    76     tcpRetransBytes     = 52932     tcpRetransSegs      =     0     tcpRetransBytes     =     0     tcpRetransSegs      =     2     tcpRetransBytes     =   752     tcpRetransSegs      =     0     tcpRetransBytes     =     0...

The following removes the drops and sets a delay of 100 milliseconds.

# ipdadm -z 73e886b7-77a5-4081-aaf6-83382e3bdced remove drop# ipdadm -z 73e886b7-77a5-4081-aaf6-83382e3bdced delay 100000#

Here's another scp to the zone with the delay set.

$ scp words root@151.1.224.26:/var/tmpwords                                         100%  202KB 28.8KB/s   00:07

Setting corruption does not work well with scp.Here is output from scp with corrupt set to 20%.

$ scp words root@151.1.224.26:/var/tmp/hash mismatchkey_verify failed for server_host_keylost connection$

Setting drop to 100% mimics the effect of a network partition, (i.e., the network looks to be down. A useful experiment to determine how well your application behaves in a network outage is to set the drop rate to 100% in the system running your SAMP or nodestack serversoftware.

There also exists a node module and library, both written by Keith Wesolowski at https://github.com/wesolows/node-ipd and https://github.com/joyent/illumos-joyent/tree/master/usr/src/lib/libipd respectively. These allow one to use ipdadm from within a Node.JS script or from within another program.

The implementation of ipdadm is in two parts: a kernel module (ipd), that gets installed as a packet filter hook (note that this refers to a paper entitled "Packet Filter Hooks Design, 3 Sep 2006", but that paper seems to have been lost to the internet) Also see Packet sniffing with packet filtering hooks; and, a user level command (ipdadm), that controls the kernel module. ipdadm is not currently in illumos, i.e., only SmartOS, and is restricted to exclusive IP stack zones (i.e., not for shared IP stacks).

Robert says it will be pushed upstream to illumos, but he's been busy. Source for the ipd kernel module is available at usr/src/uts/common/inet/ipd/. The ipd.c file in this directory has a long comment at the beginning explaining how ipd works. Source for the ipdadm user level command is at usr/src/cmd/ipdadm/.

Note that you can snoop and see dropped packets actually arriving, but outbound dropped packets are not seen. This is because ipd intercepts packets at the IP layer, not below. And, as mentioned earlier, packet drops, corruption, and delays are done full-duplex (i.e., both directions).

For information about other such enhancements that Joyent has made to SmartOS (some of which have been pushed up to illumos), see SmartOS Additions/Modifications to illumos.