tc qdisc netem notes
Usage: ... netem [ limit PACKETS ]
[ delay TIME [ JITTER [CORRELATION]]]
[ distribution {uniform|normal|pareto|paretonormal} ]
[ corrupt PERCENT [CORRELATION]]
[ duplicate PERCENT [CORRELATION]]
[ loss random PERCENT [CORRELATION]]
[ loss state P13 [P31 [P32 [P23 P14]]]
[ loss gemodel PERCENT [R [1-H [1-K]]]
[ reorder PRECENT [CORRELATION] [ gap DISTANCE ]]
[ rate RATE [PACKETOVERHEAD] [CELLSIZE] [CELLOVERHEAD]]
Show default queueing discipline
lithium:~# tc -s qdisc ls dev eth0
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 5029 bytes 21 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
pfifo_fast
Standard qdisc for 'Advanced Router' enabled kernels. Consists of a three-band queue which honors Type of Service flags,
as well as the priority that may be assigned to a packet.
Baseline Measurements:
lithium# iperf -c 172.31.1.2
------------------------------------------------------------
Client connecting to 172.31.1.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 172.31.1.1 port 44196 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 994 MBytes 834 Mbits/sec
lithium# iperf -c 172.31.1.2 -P2
------------------------------------------------------------
Client connecting to 172.31.1.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 172.31.1.1 port 44198 connected with 172.31.1.2 port 5001
[ 3] local 172.31.1.1 port 44197 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 570 MBytes 478 Mbits/sec
[ 3] 0.0-10.0 sec 548 MBytes 460 Mbits/sec
[SUM] 0.0-10.0 sec 1.09 GBytes 937 Mbits/sec
lithium# iperf -u -c 172.31.1.2 -b 1000M
------------------------------------------------------------
Client connecting to 172.31.1.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 4] local 172.31.1.1 port 36755 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 943 MBytes 791 Mbits/sec
[ 4] Sent 672968 datagrams
[ 4] Server Report:
[ 4] 0.0-10.0 sec 940 MBytes 788 Mbits/sec 0.009 ms 2631/672967 (0.39%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
lithium# iperf -u -c 172.31.1.2 -b 1000M -P2
------------------------------------------------------------
Client connecting to 172.31.1.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 5] local 172.31.1.1 port 46261 connected with 172.31.1.2 port 5001
[ 4] local 172.31.1.1 port 46175 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 570 MBytes 478 Mbits/sec
[ 5] Sent 406871 datagrams
[ 4] 0.0-10.0 sec 570 MBytes 478 Mbits/sec
[ 4] Sent 406570 datagrams
[SUM] 0.0-10.0 sec 1.11 GBytes 957 Mbits/sec
[ 5] Server Report:
[ 5] 0.0-10.0 sec 570 MBytes 478 Mbits/sec 0.019 ms 111/406870 (0.027%)
[ 5] 0.0-10.0 sec 29 datagrams received out-of-order
[ 4] Server Report:
[ 4] 0.0-10.0 sec 570 MBytes 478 Mbits/sec 0.011 ms 281/406569 (0.069%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
lithium# ping 172.31.1.2 -c3
PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data.
64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=0.444 ms
64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=0.530 ms
64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=0.515 ms
--- 172.31.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.444/0.496/0.530/0.041 ms
Configure netem based (fixed) 50ms delay on egress
lithium# tc qdisc add dev eth0 root netem delay 50ms
Measure
lithium# ping 172.31.1.2 -c3
PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data.
64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=50.6 ms
64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=50.7 ms
64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=50.7 ms
--- 172.31.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 50.600/50.698/50.772/0.197 ms
lithium# iperf -c 172.31.1.2
------------------------------------------------------------
Client connecting to 172.31.1.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 172.31.1.1 port 44199 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 170 MBytes 142 Mbits/sec
lithium# iperf -c 172.31.1.2 -P2
------------------------------------------------------------
Client connecting to 172.31.1.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 5] local 172.31.1.1 port 44201 connected with 172.31.1.2 port 5001
[ 4] local 172.31.1.1 port 44200 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 177 MBytes 148 Mbits/sec
[ 4] 0.0-10.1 sec 73.4 MBytes 61.2 Mbits/sec
[SUM] 0.0-10.1 sec 250 MBytes 209 Mbits/sec
lithium# iperf -u -c 172.31.1.2 -b 1000M
------------------------------------------------------------
Client connecting to 172.31.1.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 4] local 172.31.1.1 port 42788 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 886 MBytes 743 Mbits/sec
[ 4] Sent 631941 datagrams
[ 4] Server Report:
[ 4] 0.0-10.0 sec 273 MBytes 229 Mbits/sec 0.015 ms 437415/631940 (69%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
lithium# iperf -u -c 172.31.1.2 -b 1000M -P2
------------------------------------------------------------
Client connecting to 172.31.1.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 5] local 172.31.1.1 port 32920 connected with 172.31.1.2 port 5001
[ 4] local 172.31.1.1 port 52721 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 495 MBytes 415 Mbits/sec
[ 5] Sent 353252 datagrams
[ 5] Server Report:
[ 5] 0.0-10.2 sec 139 MBytes 114 Mbits/sec 15.732 ms 253866/352872 (72%)
[ 5] 0.0-10.2 sec 30 datagrams received out-of-order
Change the netem root qdisc to 50ms +- 25ms delay on egress
lithium# tc qdisc change dev eth0 root netem delay 50ms 25ms
Measure
lithium# ping -c6 172.31.1.2
PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data.
64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=46.2 ms
64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=62.0 ms
64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=74.2 ms
64 bytes from 172.31.1.2: icmp_req=4 ttl=64 time=75.3 ms
64 bytes from 172.31.1.2: icmp_req=5 ttl=64 time=47.5 ms
64 bytes from 172.31.1.2: icmp_req=6 ttl=64 time=57.6 ms
--- 172.31.1.2 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5008ms
rtt min/avg/max/mdev = 46.284/60.522/75.317/11.461 ms
lithium# iperf -c 172.31.1.2
------------------------------------------------------------
Client connecting to 172.31.1.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 172.31.1.1 port 44202 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 36.4 MBytes 30.5 Mbits/sec
lithium# iperf -c 172.31.1.2 -P2
------------------------------------------------------------
Client connecting to 172.31.1.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 172.31.1.1 port 44203 connected with 172.31.1.2 port 5001
[ 5] local 172.31.1.1 port 44204 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 36.1 MBytes 30.2 Mbits/sec
[ 5] 0.0-10.0 sec 25.2 MBytes 21.1 Mbits/sec
[SUM] 0.0-10.0 sec 61.4 MBytes 51.3 Mbits/sec
lithium# iperf -u -c 172.31.1.2 -b 1000M
------------------------------------------------------------
Client connecting to 172.31.1.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 4] local 172.31.1.1 port 49905 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 943 MBytes 791 Mbits/sec
[ 4] Sent 672834 datagrams
[ 4] Server Report:
[ 4] 0.0-4466.0 sec 280 MBytes 526 Kbits/sec 16.734 ms 472969/672832 (70%)
[ 4] 0.0-4466.0 sec 191972 datagrams received out-of-order
lithium# iperf -u -c 172.31.1.2 -b 1000M -P2
------------------------------------------------------------
Client connecting to 172.31.1.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 5] local 172.31.1.1 port 57588 connected with 172.31.1.2 port 5001
[ 4] local 172.31.1.1 port 50446 connected with 172.31.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 484 MBytes 406 Mbits/sec
[ 4] Sent 344967 datagrams
[ 4] Server Report:
[ 4] 0.0-4544.3 sec 141 MBytes 260 Kbits/sec 16.698 ms 244564/344965 (71%)
[ 4] 0.0-4544.3 sec 95803 datagrams received out-of-order
Lots of latency and crazy jitter, the network is on crack
lithium# tc qdisc change dev eth0 root netem delay 400ms 300ms
lithium# ping -c20 172.31.1.2
PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data.
64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=224 ms
64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=157 ms
64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=283 ms
64 bytes from 172.31.1.2: icmp_req=4 ttl=64 time=633 ms
64 bytes from 172.31.1.2: icmp_req=5 ttl=64 time=256 ms
64 bytes from 172.31.1.2: icmp_req=6 ttl=64 time=313 ms
64 bytes from 172.31.1.2: icmp_req=7 ttl=64 time=275 ms
64 bytes from 172.31.1.2: icmp_req=8 ttl=64 time=477 ms
64 bytes from 172.31.1.2: icmp_req=9 ttl=64 time=553 ms
64 bytes from 172.31.1.2: icmp_req=10 ttl=64 time=416 ms
64 bytes from 172.31.1.2: icmp_req=11 ttl=64 time=637 ms
64 bytes from 172.31.1.2: icmp_req=12 ttl=64 time=594 ms
64 bytes from 172.31.1.2: icmp_req=13 ttl=64 time=647 ms
64 bytes from 172.31.1.2: icmp_req=14 ttl=64 time=315 ms
64 bytes from 172.31.1.2: icmp_req=15 ttl=64 time=356 ms
64 bytes from 172.31.1.2: icmp_req=16 ttl=64 time=211 ms
64 bytes from 172.31.1.2: icmp_req=17 ttl=64 time=215 ms
64 bytes from 172.31.1.2: icmp_req=18 ttl=64 time=256 ms
64 bytes from 172.31.1.2: icmp_req=19 ttl=64 time=638 ms
64 bytes from 172.31.1.2: icmp_req=20 ttl=64 time=453 ms
--- 172.31.1.2 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19026ms
rtt min/avg/max/mdev = 157.775/395.904/647.801/165.088 ms
Big latency and big jitter but attempt 90% correlation of each random element with his next random element
lithium# tc qdisc change dev eth0 root netem delay 400ms 300ms 90%
lithium# ping -c20 172.31.1.2
PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data.
64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=398 ms
64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=283 ms
64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=135 ms
64 bytes from 172.31.1.2: icmp_req=4 ttl=64 time=202 ms
64 bytes from 172.31.1.2: icmp_req=5 ttl=64 time=475 ms
64 bytes from 172.31.1.2: icmp_req=6 ttl=64 time=685 ms
64 bytes from 172.31.1.2: icmp_req=7 ttl=64 time=320 ms
64 bytes from 172.31.1.2: icmp_req=8 ttl=64 time=347 ms
64 bytes from 172.31.1.2: icmp_req=9 ttl=64 time=692 ms
64 bytes from 172.31.1.2: icmp_req=10 ttl=64 time=687 ms
64 bytes from 172.31.1.2: icmp_req=11 ttl=64 time=356 ms
64 bytes from 172.31.1.2: icmp_req=12 ttl=64 time=390 ms
64 bytes from 172.31.1.2: icmp_req=13 ttl=64 time=269 ms
64 bytes from 172.31.1.2: icmp_req=14 ttl=64 time=375 ms
64 bytes from 172.31.1.2: icmp_req=15 ttl=64 time=178 ms
64 bytes from 172.31.1.2: icmp_req=16 ttl=64 time=618 ms
64 bytes from 172.31.1.2: icmp_req=17 ttl=64 time=248 ms
64 bytes from 172.31.1.2: icmp_req=18 ttl=64 time=236 ms
64 bytes from 172.31.1.2: icmp_req=19 ttl=64 time=375 ms
64 bytes from 172.31.1.2: icmp_req=20 ttl=64 time=595 ms
--- 172.31.1.2 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19010ms
rtt min/avg/max/mdev = 135.251/393.653/692.645/171.976 ms
in order to hide his crack-pipe, we may just need reduce the delay variation
lithium# tc qdisc change dev eth0 root netem delay 200ms 20ms 50%
lithium# ping -c20 172.31.1.2
PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data.
64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=206 ms
64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=196 ms
64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=196 ms
64 bytes from 172.31.1.2: icmp_req=4 ttl=64 time=211 ms
64 bytes from 172.31.1.2: icmp_req=5 ttl=64 time=191 ms
64 bytes from 172.31.1.2: icmp_req=6 ttl=64 time=188 ms
64 bytes from 172.31.1.2: icmp_req=7 ttl=64 time=213 ms
64 bytes from 172.31.1.2: icmp_req=8 ttl=64 time=219 ms
64 bytes from 172.31.1.2: icmp_req=9 ttl=64 time=185 ms
64 bytes from 172.31.1.2: icmp_req=10 ttl=64 time=196 ms
64 bytes from 172.31.1.2: icmp_req=11 ttl=64 time=193 ms
64 bytes from 172.31.1.2: icmp_req=12 ttl=64 time=194 ms
64 bytes from 172.31.1.2: icmp_req=13 ttl=64 time=215 ms
64 bytes from 172.31.1.2: icmp_req=14 ttl=64 time=209 ms
64 bytes from 172.31.1.2: icmp_req=15 ttl=64 time=184 ms
64 bytes from 172.31.1.2: icmp_req=16 ttl=64 time=182 ms
64 bytes from 172.31.1.2: icmp_req=17 ttl=64 time=183 ms
64 bytes from 172.31.1.2: icmp_req=18 ttl=64 time=213 ms
64 bytes from 172.31.1.2: icmp_req=19 ttl=64 time=187 ms
64 bytes from 172.31.1.2: icmp_req=20 ttl=64 time=207 ms
--- 172.31.1.2 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19028ms
rtt min/avg/max/mdev = 182.506/198.908/219.301/11.804 ms
See qdisc statistics
lithium# tc -s qdisc ls dev eth0
qdisc netem 8005: root refcnt 2 limit 1000 delay 200.0ms 20.0ms 50%
Sent 2367236851 bytes 1567250 pkt (dropped 2837991, overlimits 0 requeues 0)
backlog 106b 1p requeues 0
Use a normal distribution for the pseudorandom delay variation
lithium# tc qdisc change dev eth0 root netem delay 200ms 20ms distribution normal
lithium# ping -c20 172.31.1.2
PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data.
64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=195 ms
64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=196 ms
64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=183 ms
64 bytes from 172.31.1.2: icmp_req=4 ttl=64 time=183 ms
64 bytes from 172.31.1.2: icmp_req=5 ttl=64 time=166 ms
64 bytes from 172.31.1.2: icmp_req=6 ttl=64 time=208 ms
64 bytes from 172.31.1.2: icmp_req=7 ttl=64 time=224 ms
64 bytes from 172.31.1.2: icmp_req=8 ttl=64 time=216 ms
64 bytes from 172.31.1.2: icmp_req=9 ttl=64 time=211 ms
64 bytes from 172.31.1.2: icmp_req=10 ttl=64 time=221 ms
64 bytes from 172.31.1.2: icmp_req=11 ttl=64 time=206 ms
64 bytes from 172.31.1.2: icmp_req=12 ttl=64 time=219 ms
64 bytes from 172.31.1.2: icmp_req=13 ttl=64 time=214 ms
64 bytes from 172.31.1.2: icmp_req=14 ttl=64 time=197 ms
64 bytes from 172.31.1.2: icmp_req=15 ttl=64 time=216 ms
64 bytes from 172.31.1.2: icmp_req=16 ttl=64 time=182 ms
64 bytes from 172.31.1.2: icmp_req=17 ttl=64 time=179 ms
64 bytes from 172.31.1.2: icmp_req=18 ttl=64 time=223 ms
64 bytes from 172.31.1.2: icmp_req=19 ttl=64 time=209 ms
64 bytes from 172.31.1.2: icmp_req=20 ttl=64 time=220 ms
--- 172.31.1.2 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19030ms
rtt min/avg/max/mdev = 166.480/203.929/224.446/16.774 ms
... pinging lithium from another host we should observe a similar behavior but now his ECHO_REPLYs are slowed down
beryllium# ping -A -c20 172.31.1.1
PING 172.31.1.1 (172.31.1.1) 56(84) bytes of data.
64 bytes from 172.31.1.1: icmp_req=1 ttl=64 time=203 ms
64 bytes from 172.31.1.1: icmp_req=2 ttl=64 time=204 ms
64 bytes from 172.31.1.1: icmp_req=3 ttl=64 time=207 ms
64 bytes from 172.31.1.1: icmp_req=4 ttl=64 time=221 ms
64 bytes from 172.31.1.1: icmp_req=5 ttl=64 time=185 ms
64 bytes from 172.31.1.1: icmp_req=6 ttl=64 time=224 ms
64 bytes from 172.31.1.1: icmp_req=7 ttl=64 time=182 ms
64 bytes from 172.31.1.1: icmp_req=8 ttl=64 time=178 ms
64 bytes from 172.31.1.1: icmp_req=9 ttl=64 time=220 ms
64 bytes from 172.31.1.1: icmp_req=10 ttl=64 time=236 ms
64 bytes from 172.31.1.1: icmp_req=11 ttl=64 time=180 ms
64 bytes from 172.31.1.1: icmp_req=12 ttl=64 time=184 ms
64 bytes from 172.31.1.1: icmp_req=13 ttl=64 time=205 ms
64 bytes from 172.31.1.1: icmp_req=14 ttl=64 time=197 ms
64 bytes from 172.31.1.1: icmp_req=15 ttl=64 time=227 ms
64 bytes from 172.31.1.1: icmp_req=16 ttl=64 time=173 ms
64 bytes from 172.31.1.1: icmp_req=17 ttl=64 time=180 ms
64 bytes from 172.31.1.1: icmp_req=18 ttl=64 time=199 ms
64 bytes from 172.31.1.1: icmp_req=19 ttl=64 time=192 ms
64 bytes from 172.31.1.1: icmp_req=20 ttl=64 time=204 ms
--- 172.31.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 3842ms
rtt min/avg/max/mdev = 173.172/200.557/236.417/17.955 ms, pipe 2, ipg/ewma 202.210/198.474 ms
Availabe probability distributions
lithium# ls -1 /usr/lib/tc/*dist
/usr/lib/tc/experimental.dist
/usr/lib/tc/normal.dist
/usr/lib/tc/pareto.dist
/usr/lib/tc/paretonormal.dist
Drop random 0.5% of packets ( tc, not layer3, packets )
lithium# tc qdisc change dev eth0 root netem loss 0.5%
lithium# ping -A -c10000 -q beryllium
PING beryllium (172.31.1.2) 56(84) bytes of data.
--- beryllium ping statistics ---
10000 packets transmitted, 9948 received, 0% packet loss, time 3538ms
rtt min/avg/max/mdev = 0.065/0.159/3.710/0.094 ms, ipg/ewma 0.353/0.145 ms
beryllium# ping -A -c10000 -q lithium
PING lithium (172.31.1.1) 56(84) bytes of data.
--- lithium ping statistics ---
10000 packets transmitted, 9949 received, 0% packet loss, time 3470ms
rtt min/avg/max/mdev = 0.069/0.166/3.337/0.088 ms, ipg/ewma 0.347/0.156 ms
Show qdisc statistics
lithium# tc -s qdisc ls dev eth0
qdisc netem 8007: root refcnt 2 limit 1000 loss 0.5%
Sent 1962385 bytes 20015 pkt (dropped 103, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
Remove our netem queing discipline
lithium# tc qdisc del dev eth0 root
Baseline measurements
lithium# ping -A -c10000 -q beryllium
PING beryllium (172.31.1.2) 56(84) bytes of data.
--- beryllium ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 2730ms
rtt min/avg/max/mdev = 0.062/0.146/0.602/0.079 ms, ipg/ewma 0.273/0.117 ms
beryllium# ping -A -c10000 -q lithium
PING lithium (172.31.1.1) 56(84) bytes of data.
--- lithium ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 2868ms
rtt min/avg/max/mdev = 0.066/0.163/3.487/0.091 ms, ipg/ewma 0.286/0.220 ms
1% (random) drop with probality(n) = 0.8 * propablity(n-1) + 0.2 * Random
lithium# tc qdisc add dev eth0 root netem loss 1% 80%
skipping the measurements --they are inconclusive
... hmm , maybe I need to put sodium --one more host-- in the game
~3% of duplicates
lithium# tc qdisc change dev eth0 root netem duplicate 3%
lithium# ping -A -c10000 -q beryllium
PING beryllium (172.31.1.2) 56(84) bytes of data.
--- beryllium ping statistics ---
10000 packets transmitted, 10000 received, +317 duplicates, 0% packet loss, time 2844ms
rtt min/avg/max/mdev = 0.066/0.149/3.409/0.075 ms, pipe 2, ipg/ewma 0.284/0.131 ms
~1% of single bit errors in the packets ( frames? )
lithium# tc qdisc change dev eth0 root netem corrupt 1%
lithium# ping -A -c10000 -q beryllium
PING beryllium (172.31.1.2) 56(84) bytes of data.
--- beryllium ping statistics ---
10000 packets transmitted, 9909 received, 0% packet loss, time 3874ms
rtt min/avg/max/mdev = 0.064/0.156/0.648/0.078 ms, ipg/ewma 0.387/0.165 ms
force out of order with gap
gap uses a fixed sequence and reorders every Nth packet
eg the following should cause ~20% of every 3rd packet to be sent immediately while the ~previous two packets are delayed for 10ms
lithium# tc qdisc change dev eth0 root netem reorder 20% gap 3 delay 10ms
lithium# ping -A -c10000 -q beryllium
PING beryllium (172.31.1.2) 56(84) bytes of data.
--- beryllium ping statistics ---
10000 packets transmitted, 9895 received, 1% packet loss, time 102198ms
rtt min/avg/max/mdev = 0.142/9.139/13.900/3.517 ms, pipe 2, ipg/ewma 10.220/8.481 ms
lithium# tc -s qdisc ls dev eth0
qdisc netem 8008: root refcnt 2 limit 1000 delay 10.0ms reorder 20% corrupt 1% gap 3
Sent 2009781566 bytes 1745324 pkt (dropped 10, overlimits 0 requeues 2653)
backlog 0b 0p requeues 2653
Force out of order
25% of packets (with a correlation of 50%) will get sent immediately, (all) others will be delayed by 10ms
lithium# tc qdisc change dev eth0 root netem delay 10ms reorder 25% 50%
lithium# ping -A -c10000 -q beryllium
PING beryllium (172.31.1.2) 56(84) bytes of data.
--- beryllium ping statistics ---
10000 packets transmitted, 9899 received, 1% packet loss, time 104680ms
rtt min/avg/max/mdev = 0.109/9.829/13.955/2.632 ms, pipe 2, ipg/ewma 10.469/7.666 ms
lithium# ping -A -c10 beryllium
PING beryllium (172.31.1.2) 56(84) bytes of data.
64 bytes from beryllium (172.31.1.2): icmp_req=1 ttl=64 time=10.6 ms
64 bytes from beryllium (172.31.1.2): icmp_req=2 ttl=64 time=10.5 ms
64 bytes from beryllium (172.31.1.2): icmp_req=3 ttl=64 time=10.5 ms
64 bytes from beryllium (172.31.1.2): icmp_req=4 ttl=64 time=10.4 ms
64 bytes from beryllium (172.31.1.2): icmp_req=5 ttl=64 time=10.4 ms
64 bytes from beryllium (172.31.1.2): icmp_req=6 ttl=64 time=10.6 ms
64 bytes from beryllium (172.31.1.2): icmp_req=7 ttl=64 time=10.5 ms
64 bytes from beryllium (172.31.1.2): icmp_req=8 ttl=64 time=10.6 ms
64 bytes from beryllium (172.31.1.2): icmp_req=9 ttl=64 time=10.8 ms
64 bytes from beryllium (172.31.1.2): icmp_req=10 ttl=64 time=10.5 ms
--- beryllium ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 100ms
rtt min/avg/max/mdev = 10.486/10.582/10.813/0.150 ms, ipg/ewma 11.116/10.600 ms
Reorder with delay
eg delay each packet 100ms +-rand(75)ms Jitter
lithium# tc qdisc change dev eth0 root netem delay 100ms 75ms
lithium# ping -A -c10 beryllium
PING beryllium (172.31.1.2) 56(84) bytes of data.
64 bytes from beryllium (172.31.1.2): icmp_req=1 ttl=64 time=152 ms
64 bytes from beryllium (172.31.1.2): icmp_req=2 ttl=64 time=168 ms
64 bytes from beryllium (172.31.1.2): icmp_req=3 ttl=64 time=88.1 ms
64 bytes from beryllium (172.31.1.2): icmp_req=4 ttl=64 time=137 ms
64 bytes from beryllium (172.31.1.2): icmp_req=5 ttl=64 time=145 ms
64 bytes from beryllium (172.31.1.2): icmp_req=6 ttl=64 time=105 ms
64 bytes from beryllium (172.31.1.2): icmp_req=7 ttl=64 time=138 ms
64 bytes from beryllium (172.31.1.2): icmp_req=8 ttl=64 time=164 ms
64 bytes from beryllium (172.31.1.2): icmp_req=9 ttl=64 time=174 ms
64 bytes from beryllium (172.31.1.2): icmp_req=10 ttl=64 time=123 ms
--- beryllium ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 1307ms
rtt min/avg/max/mdev = 88.111/139.794/174.547/26.414 ms, pipe 2, ipg/ewma 145.244/143.915 ms
lithium# ping -A -c1000 -q beryllium
PING beryllium (172.31.1.2) 56(84) bytes of data.
--- beryllium ping statistics ---
1000 packets transmitted, 991 received, 0% packet loss, time 102193ms
rtt min/avg/max/mdev = 26.115/102.891/175.749/43.053 ms, pipe 3, ipg/ewma 102.295/92.228 ms
netem
alog netem
