Usage: ... netem [ limit PACKETS ]
[ delay TIME [ JITTER [CORRELATION]]]
[ distribution {uniform|normal|pareto|paretonormal} ]
[ corrupt PERCENT [CORRELATION]]
[ duplicate PERCENT [CORRELATION]]
[ loss random PERCENT [CORRELATION]]
[ loss state P13 [P31 [P32 [P23 P14]]]
[ loss gemodel PERCENT [R [1-H [1-K]]]
[ reorder PRECENT [CORRELATION] [ gap DISTANCE ]]
[ rate RATE [PACKETOVERHEAD] [CELLSIZE] [CELLOVERHEAD]]
Show default queueing discipline
lithium:~# tc -s qdisc ls dev eth0 qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 5029 bytes 21 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0
pfifo_fast
Standard qdisc for 'Advanced Router' enabled kernels. Consists of a three-band queue which honors Type of Service flags,
as well as the priority that may be assigned to a packet.
Baseline Measurements:
lithium# iperf -c 172.31.1.2 ------------------------------------------------------------ Client connecting to 172.31.1.2, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 172.31.1.1 port 44196 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 994 MBytes 834 Mbits/sec lithium# iperf -c 172.31.1.2 -P2 ------------------------------------------------------------ Client connecting to 172.31.1.2, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 4] local 172.31.1.1 port 44198 connected with 172.31.1.2 port 5001 [ 3] local 172.31.1.1 port 44197 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 570 MBytes 478 Mbits/sec [ 3] 0.0-10.0 sec 548 MBytes 460 Mbits/sec [SUM] 0.0-10.0 sec 1.09 GBytes 937 Mbits/sec
lithium# iperf -u -c 172.31.1.2 -b 1000M ------------------------------------------------------------ Client connecting to 172.31.1.2, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 4] local 172.31.1.1 port 36755 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 943 MBytes 791 Mbits/sec [ 4] Sent 672968 datagrams [ 4] Server Report: [ 4] 0.0-10.0 sec 940 MBytes 788 Mbits/sec 0.009 ms 2631/672967 (0.39%) [ 4] 0.0-10.0 sec 1 datagrams received out-of-order lithium# iperf -u -c 172.31.1.2 -b 1000M -P2 ------------------------------------------------------------ Client connecting to 172.31.1.2, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 5] local 172.31.1.1 port 46261 connected with 172.31.1.2 port 5001 [ 4] local 172.31.1.1 port 46175 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 5] 0.0-10.0 sec 570 MBytes 478 Mbits/sec [ 5] Sent 406871 datagrams [ 4] 0.0-10.0 sec 570 MBytes 478 Mbits/sec [ 4] Sent 406570 datagrams [SUM] 0.0-10.0 sec 1.11 GBytes 957 Mbits/sec [ 5] Server Report: [ 5] 0.0-10.0 sec 570 MBytes 478 Mbits/sec 0.019 ms 111/406870 (0.027%) [ 5] 0.0-10.0 sec 29 datagrams received out-of-order [ 4] Server Report: [ 4] 0.0-10.0 sec 570 MBytes 478 Mbits/sec 0.011 ms 281/406569 (0.069%) [ 4] 0.0-10.0 sec 1 datagrams received out-of-order
lithium# ping 172.31.1.2 -c3 PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data. 64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=0.444 ms 64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=0.530 ms 64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=0.515 ms --- 172.31.1.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.444/0.496/0.530/0.041 ms
Configure netem based (fixed) 50ms delay on egress
lithium# tc qdisc add dev eth0 root netem delay 50msMeasure
lithium# ping 172.31.1.2 -c3 PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data. 64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=50.6 ms 64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=50.7 ms 64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=50.7 ms --- 172.31.1.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 50.600/50.698/50.772/0.197 ms
lithium# iperf -c 172.31.1.2 ------------------------------------------------------------ Client connecting to 172.31.1.2, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 4] local 172.31.1.1 port 44199 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 170 MBytes 142 Mbits/sec lithium# iperf -c 172.31.1.2 -P2 ------------------------------------------------------------ Client connecting to 172.31.1.2, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 5] local 172.31.1.1 port 44201 connected with 172.31.1.2 port 5001 [ 4] local 172.31.1.1 port 44200 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 5] 0.0-10.0 sec 177 MBytes 148 Mbits/sec [ 4] 0.0-10.1 sec 73.4 MBytes 61.2 Mbits/sec [SUM] 0.0-10.1 sec 250 MBytes 209 Mbits/sec
lithium# iperf -u -c 172.31.1.2 -b 1000M ------------------------------------------------------------ Client connecting to 172.31.1.2, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 4] local 172.31.1.1 port 42788 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 886 MBytes 743 Mbits/sec [ 4] Sent 631941 datagrams [ 4] Server Report: [ 4] 0.0-10.0 sec 273 MBytes 229 Mbits/sec 0.015 ms 437415/631940 (69%) [ 4] 0.0-10.0 sec 1 datagrams received out-of-order lithium# iperf -u -c 172.31.1.2 -b 1000M -P2 ------------------------------------------------------------ Client connecting to 172.31.1.2, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 5] local 172.31.1.1 port 32920 connected with 172.31.1.2 port 5001 [ 4] local 172.31.1.1 port 52721 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 5] 0.0-10.0 sec 495 MBytes 415 Mbits/sec [ 5] Sent 353252 datagrams [ 5] Server Report: [ 5] 0.0-10.2 sec 139 MBytes 114 Mbits/sec 15.732 ms 253866/352872 (72%) [ 5] 0.0-10.2 sec 30 datagrams received out-of-order
Change the netem root qdisc to 50ms +- 25ms delay on egress
lithium# tc qdisc change dev eth0 root netem delay 50ms 25ms
Measure
lithium# ping -c6 172.31.1.2 PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data. 64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=46.2 ms 64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=62.0 ms 64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=74.2 ms 64 bytes from 172.31.1.2: icmp_req=4 ttl=64 time=75.3 ms 64 bytes from 172.31.1.2: icmp_req=5 ttl=64 time=47.5 ms 64 bytes from 172.31.1.2: icmp_req=6 ttl=64 time=57.6 ms --- 172.31.1.2 ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 5008ms rtt min/avg/max/mdev = 46.284/60.522/75.317/11.461 ms
lithium# iperf -c 172.31.1.2 ------------------------------------------------------------ Client connecting to 172.31.1.2, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 4] local 172.31.1.1 port 44202 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 36.4 MBytes 30.5 Mbits/sec lithium# iperf -c 172.31.1.2 -P2 ------------------------------------------------------------ Client connecting to 172.31.1.2, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 4] local 172.31.1.1 port 44203 connected with 172.31.1.2 port 5001 [ 5] local 172.31.1.1 port 44204 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 36.1 MBytes 30.2 Mbits/sec [ 5] 0.0-10.0 sec 25.2 MBytes 21.1 Mbits/sec [SUM] 0.0-10.0 sec 61.4 MBytes 51.3 Mbits/sec lithium# iperf -u -c 172.31.1.2 -b 1000M ------------------------------------------------------------ Client connecting to 172.31.1.2, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 4] local 172.31.1.1 port 49905 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 943 MBytes 791 Mbits/sec [ 4] Sent 672834 datagrams [ 4] Server Report: [ 4] 0.0-4466.0 sec 280 MBytes 526 Kbits/sec 16.734 ms 472969/672832 (70%) [ 4] 0.0-4466.0 sec 191972 datagrams received out-of-order lithium# iperf -u -c 172.31.1.2 -b 1000M -P2 ------------------------------------------------------------ Client connecting to 172.31.1.2, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 5] local 172.31.1.1 port 57588 connected with 172.31.1.2 port 5001 [ 4] local 172.31.1.1 port 50446 connected with 172.31.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 484 MBytes 406 Mbits/sec [ 4] Sent 344967 datagrams [ 4] Server Report: [ 4] 0.0-4544.3 sec 141 MBytes 260 Kbits/sec 16.698 ms 244564/344965 (71%) [ 4] 0.0-4544.3 sec 95803 datagrams received out-of-order
Lots of latency and crazy jitter, the network is on crack
lithium# tc qdisc change dev eth0 root netem delay 400ms 300ms lithium# ping -c20 172.31.1.2 PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data. 64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=224 ms 64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=157 ms 64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=283 ms 64 bytes from 172.31.1.2: icmp_req=4 ttl=64 time=633 ms 64 bytes from 172.31.1.2: icmp_req=5 ttl=64 time=256 ms 64 bytes from 172.31.1.2: icmp_req=6 ttl=64 time=313 ms 64 bytes from 172.31.1.2: icmp_req=7 ttl=64 time=275 ms 64 bytes from 172.31.1.2: icmp_req=8 ttl=64 time=477 ms 64 bytes from 172.31.1.2: icmp_req=9 ttl=64 time=553 ms 64 bytes from 172.31.1.2: icmp_req=10 ttl=64 time=416 ms 64 bytes from 172.31.1.2: icmp_req=11 ttl=64 time=637 ms 64 bytes from 172.31.1.2: icmp_req=12 ttl=64 time=594 ms 64 bytes from 172.31.1.2: icmp_req=13 ttl=64 time=647 ms 64 bytes from 172.31.1.2: icmp_req=14 ttl=64 time=315 ms 64 bytes from 172.31.1.2: icmp_req=15 ttl=64 time=356 ms 64 bytes from 172.31.1.2: icmp_req=16 ttl=64 time=211 ms 64 bytes from 172.31.1.2: icmp_req=17 ttl=64 time=215 ms 64 bytes from 172.31.1.2: icmp_req=18 ttl=64 time=256 ms 64 bytes from 172.31.1.2: icmp_req=19 ttl=64 time=638 ms 64 bytes from 172.31.1.2: icmp_req=20 ttl=64 time=453 ms --- 172.31.1.2 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 19026ms rtt min/avg/max/mdev = 157.775/395.904/647.801/165.088 ms
Big latency and big jitter but attempt 90% correlation of each random element with his next random element
lithium# tc qdisc change dev eth0 root netem delay 400ms 300ms 90% lithium# ping -c20 172.31.1.2 PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data. 64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=398 ms 64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=283 ms 64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=135 ms 64 bytes from 172.31.1.2: icmp_req=4 ttl=64 time=202 ms 64 bytes from 172.31.1.2: icmp_req=5 ttl=64 time=475 ms 64 bytes from 172.31.1.2: icmp_req=6 ttl=64 time=685 ms 64 bytes from 172.31.1.2: icmp_req=7 ttl=64 time=320 ms 64 bytes from 172.31.1.2: icmp_req=8 ttl=64 time=347 ms 64 bytes from 172.31.1.2: icmp_req=9 ttl=64 time=692 ms 64 bytes from 172.31.1.2: icmp_req=10 ttl=64 time=687 ms 64 bytes from 172.31.1.2: icmp_req=11 ttl=64 time=356 ms 64 bytes from 172.31.1.2: icmp_req=12 ttl=64 time=390 ms 64 bytes from 172.31.1.2: icmp_req=13 ttl=64 time=269 ms 64 bytes from 172.31.1.2: icmp_req=14 ttl=64 time=375 ms 64 bytes from 172.31.1.2: icmp_req=15 ttl=64 time=178 ms 64 bytes from 172.31.1.2: icmp_req=16 ttl=64 time=618 ms 64 bytes from 172.31.1.2: icmp_req=17 ttl=64 time=248 ms 64 bytes from 172.31.1.2: icmp_req=18 ttl=64 time=236 ms 64 bytes from 172.31.1.2: icmp_req=19 ttl=64 time=375 ms 64 bytes from 172.31.1.2: icmp_req=20 ttl=64 time=595 ms --- 172.31.1.2 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 19010ms rtt min/avg/max/mdev = 135.251/393.653/692.645/171.976 ms
in order to hide his crack-pipe, we may just need reduce the delay variation
lithium# tc qdisc change dev eth0 root netem delay 200ms 20ms 50% lithium# ping -c20 172.31.1.2 PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data. 64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=206 ms 64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=196 ms 64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=196 ms 64 bytes from 172.31.1.2: icmp_req=4 ttl=64 time=211 ms 64 bytes from 172.31.1.2: icmp_req=5 ttl=64 time=191 ms 64 bytes from 172.31.1.2: icmp_req=6 ttl=64 time=188 ms 64 bytes from 172.31.1.2: icmp_req=7 ttl=64 time=213 ms 64 bytes from 172.31.1.2: icmp_req=8 ttl=64 time=219 ms 64 bytes from 172.31.1.2: icmp_req=9 ttl=64 time=185 ms 64 bytes from 172.31.1.2: icmp_req=10 ttl=64 time=196 ms 64 bytes from 172.31.1.2: icmp_req=11 ttl=64 time=193 ms 64 bytes from 172.31.1.2: icmp_req=12 ttl=64 time=194 ms 64 bytes from 172.31.1.2: icmp_req=13 ttl=64 time=215 ms 64 bytes from 172.31.1.2: icmp_req=14 ttl=64 time=209 ms 64 bytes from 172.31.1.2: icmp_req=15 ttl=64 time=184 ms 64 bytes from 172.31.1.2: icmp_req=16 ttl=64 time=182 ms 64 bytes from 172.31.1.2: icmp_req=17 ttl=64 time=183 ms 64 bytes from 172.31.1.2: icmp_req=18 ttl=64 time=213 ms 64 bytes from 172.31.1.2: icmp_req=19 ttl=64 time=187 ms 64 bytes from 172.31.1.2: icmp_req=20 ttl=64 time=207 ms --- 172.31.1.2 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 19028ms rtt min/avg/max/mdev = 182.506/198.908/219.301/11.804 ms
See qdisc statistics
lithium# tc -s qdisc ls dev eth0 qdisc netem 8005: root refcnt 2 limit 1000 delay 200.0ms 20.0ms 50% Sent 2367236851 bytes 1567250 pkt (dropped 2837991, overlimits 0 requeues 0) backlog 106b 1p requeues 0
Use a normal distribution for the pseudorandom delay variation
lithium# tc qdisc change dev eth0 root netem delay 200ms 20ms distribution normal lithium# ping -c20 172.31.1.2 PING 172.31.1.2 (172.31.1.2) 56(84) bytes of data. 64 bytes from 172.31.1.2: icmp_req=1 ttl=64 time=195 ms 64 bytes from 172.31.1.2: icmp_req=2 ttl=64 time=196 ms 64 bytes from 172.31.1.2: icmp_req=3 ttl=64 time=183 ms 64 bytes from 172.31.1.2: icmp_req=4 ttl=64 time=183 ms 64 bytes from 172.31.1.2: icmp_req=5 ttl=64 time=166 ms 64 bytes from 172.31.1.2: icmp_req=6 ttl=64 time=208 ms 64 bytes from 172.31.1.2: icmp_req=7 ttl=64 time=224 ms 64 bytes from 172.31.1.2: icmp_req=8 ttl=64 time=216 ms 64 bytes from 172.31.1.2: icmp_req=9 ttl=64 time=211 ms 64 bytes from 172.31.1.2: icmp_req=10 ttl=64 time=221 ms 64 bytes from 172.31.1.2: icmp_req=11 ttl=64 time=206 ms 64 bytes from 172.31.1.2: icmp_req=12 ttl=64 time=219 ms 64 bytes from 172.31.1.2: icmp_req=13 ttl=64 time=214 ms 64 bytes from 172.31.1.2: icmp_req=14 ttl=64 time=197 ms 64 bytes from 172.31.1.2: icmp_req=15 ttl=64 time=216 ms 64 bytes from 172.31.1.2: icmp_req=16 ttl=64 time=182 ms 64 bytes from 172.31.1.2: icmp_req=17 ttl=64 time=179 ms 64 bytes from 172.31.1.2: icmp_req=18 ttl=64 time=223 ms 64 bytes from 172.31.1.2: icmp_req=19 ttl=64 time=209 ms 64 bytes from 172.31.1.2: icmp_req=20 ttl=64 time=220 ms --- 172.31.1.2 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 19030ms rtt min/avg/max/mdev = 166.480/203.929/224.446/16.774 ms
... pinging lithium from another host we should observe a similar behavior but now his ECHO_REPLYs are slowed down
beryllium# ping -A -c20 172.31.1.1 PING 172.31.1.1 (172.31.1.1) 56(84) bytes of data. 64 bytes from 172.31.1.1: icmp_req=1 ttl=64 time=203 ms 64 bytes from 172.31.1.1: icmp_req=2 ttl=64 time=204 ms 64 bytes from 172.31.1.1: icmp_req=3 ttl=64 time=207 ms 64 bytes from 172.31.1.1: icmp_req=4 ttl=64 time=221 ms 64 bytes from 172.31.1.1: icmp_req=5 ttl=64 time=185 ms 64 bytes from 172.31.1.1: icmp_req=6 ttl=64 time=224 ms 64 bytes from 172.31.1.1: icmp_req=7 ttl=64 time=182 ms 64 bytes from 172.31.1.1: icmp_req=8 ttl=64 time=178 ms 64 bytes from 172.31.1.1: icmp_req=9 ttl=64 time=220 ms 64 bytes from 172.31.1.1: icmp_req=10 ttl=64 time=236 ms 64 bytes from 172.31.1.1: icmp_req=11 ttl=64 time=180 ms 64 bytes from 172.31.1.1: icmp_req=12 ttl=64 time=184 ms 64 bytes from 172.31.1.1: icmp_req=13 ttl=64 time=205 ms 64 bytes from 172.31.1.1: icmp_req=14 ttl=64 time=197 ms 64 bytes from 172.31.1.1: icmp_req=15 ttl=64 time=227 ms 64 bytes from 172.31.1.1: icmp_req=16 ttl=64 time=173 ms 64 bytes from 172.31.1.1: icmp_req=17 ttl=64 time=180 ms 64 bytes from 172.31.1.1: icmp_req=18 ttl=64 time=199 ms 64 bytes from 172.31.1.1: icmp_req=19 ttl=64 time=192 ms 64 bytes from 172.31.1.1: icmp_req=20 ttl=64 time=204 ms --- 172.31.1.1 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 3842ms rtt min/avg/max/mdev = 173.172/200.557/236.417/17.955 ms, pipe 2, ipg/ewma 202.210/198.474 ms
Availabe probability distributions
lithium# ls -1 /usr/lib/tc/*dist /usr/lib/tc/experimental.dist /usr/lib/tc/normal.dist /usr/lib/tc/pareto.dist /usr/lib/tc/paretonormal.dist
Drop random 0.5% of packets ( tc, not layer3, packets )
lithium# tc qdisc change dev eth0 root netem loss 0.5% lithium# ping -A -c10000 -q beryllium PING beryllium (172.31.1.2) 56(84) bytes of data. --- beryllium ping statistics --- 10000 packets transmitted, 9948 received, 0% packet loss, time 3538ms rtt min/avg/max/mdev = 0.065/0.159/3.710/0.094 ms, ipg/ewma 0.353/0.145 ms
beryllium# ping -A -c10000 -q lithium PING lithium (172.31.1.1) 56(84) bytes of data. --- lithium ping statistics --- 10000 packets transmitted, 9949 received, 0% packet loss, time 3470ms rtt min/avg/max/mdev = 0.069/0.166/3.337/0.088 ms, ipg/ewma 0.347/0.156 ms
Show qdisc statistics
lithium# tc -s qdisc ls dev eth0 qdisc netem 8007: root refcnt 2 limit 1000 loss 0.5% Sent 1962385 bytes 20015 pkt (dropped 103, overlimits 0 requeues 0) backlog 0b 0p requeues 0
Remove our netem queing discipline
lithium# tc qdisc del dev eth0 root
Baseline measurements
lithium# ping -A -c10000 -q beryllium PING beryllium (172.31.1.2) 56(84) bytes of data. --- beryllium ping statistics --- 10000 packets transmitted, 10000 received, 0% packet loss, time 2730ms rtt min/avg/max/mdev = 0.062/0.146/0.602/0.079 ms, ipg/ewma 0.273/0.117 ms
beryllium# ping -A -c10000 -q lithium PING lithium (172.31.1.1) 56(84) bytes of data. --- lithium ping statistics --- 10000 packets transmitted, 10000 received, 0% packet loss, time 2868ms rtt min/avg/max/mdev = 0.066/0.163/3.487/0.091 ms, ipg/ewma 0.286/0.220 ms
1% (random) drop with probality(n) = 0.8 * propablity(n-1) + 0.2 * Random
lithium# tc qdisc add dev eth0 root netem loss 1% 80%
skipping the measurements --they are inconclusive
... hmm , maybe I need to put sodium --one more host-- in the game
~3% of duplicates
lithium# tc qdisc change dev eth0 root netem duplicate 3% lithium# ping -A -c10000 -q beryllium PING beryllium (172.31.1.2) 56(84) bytes of data. --- beryllium ping statistics --- 10000 packets transmitted, 10000 received, +317 duplicates, 0% packet loss, time 2844ms rtt min/avg/max/mdev = 0.066/0.149/3.409/0.075 ms, pipe 2, ipg/ewma 0.284/0.131 ms
~1% of single bit errors in the packets ( frames? )
lithium# tc qdisc change dev eth0 root netem corrupt 1% lithium# ping -A -c10000 -q beryllium PING beryllium (172.31.1.2) 56(84) bytes of data. --- beryllium ping statistics --- 10000 packets transmitted, 9909 received, 0% packet loss, time 3874ms rtt min/avg/max/mdev = 0.064/0.156/0.648/0.078 ms, ipg/ewma 0.387/0.165 ms
force out of order with gap
gap uses a fixed sequence and reorders every Nth packet
eg the following should cause ~20% of every 3rd packet to be sent immediately while the ~previous two packets are delayed for 10ms
lithium# tc qdisc change dev eth0 root netem reorder 20% gap 3 delay 10ms lithium# ping -A -c10000 -q beryllium PING beryllium (172.31.1.2) 56(84) bytes of data. --- beryllium ping statistics --- 10000 packets transmitted, 9895 received, 1% packet loss, time 102198ms rtt min/avg/max/mdev = 0.142/9.139/13.900/3.517 ms, pipe 2, ipg/ewma 10.220/8.481 ms lithium# tc -s qdisc ls dev eth0 qdisc netem 8008: root refcnt 2 limit 1000 delay 10.0ms reorder 20% corrupt 1% gap 3 Sent 2009781566 bytes 1745324 pkt (dropped 10, overlimits 0 requeues 2653) backlog 0b 0p requeues 2653
Force out of order
25% of packets (with a correlation of 50%) will get sent immediately, (all) others will be delayed by 10ms
lithium# tc qdisc change dev eth0 root netem delay 10ms reorder 25% 50% lithium# ping -A -c10000 -q beryllium PING beryllium (172.31.1.2) 56(84) bytes of data. --- beryllium ping statistics --- 10000 packets transmitted, 9899 received, 1% packet loss, time 104680ms rtt min/avg/max/mdev = 0.109/9.829/13.955/2.632 ms, pipe 2, ipg/ewma 10.469/7.666 ms lithium# ping -A -c10 beryllium PING beryllium (172.31.1.2) 56(84) bytes of data. 64 bytes from beryllium (172.31.1.2): icmp_req=1 ttl=64 time=10.6 ms 64 bytes from beryllium (172.31.1.2): icmp_req=2 ttl=64 time=10.5 ms 64 bytes from beryllium (172.31.1.2): icmp_req=3 ttl=64 time=10.5 ms 64 bytes from beryllium (172.31.1.2): icmp_req=4 ttl=64 time=10.4 ms 64 bytes from beryllium (172.31.1.2): icmp_req=5 ttl=64 time=10.4 ms 64 bytes from beryllium (172.31.1.2): icmp_req=6 ttl=64 time=10.6 ms 64 bytes from beryllium (172.31.1.2): icmp_req=7 ttl=64 time=10.5 ms 64 bytes from beryllium (172.31.1.2): icmp_req=8 ttl=64 time=10.6 ms 64 bytes from beryllium (172.31.1.2): icmp_req=9 ttl=64 time=10.8 ms 64 bytes from beryllium (172.31.1.2): icmp_req=10 ttl=64 time=10.5 ms --- beryllium ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 100ms rtt min/avg/max/mdev = 10.486/10.582/10.813/0.150 ms, ipg/ewma 11.116/10.600 ms
Reorder with delay eg delay each packet 100ms +-rand(75)ms Jitter
lithium# tc qdisc change dev eth0 root netem delay 100ms 75ms lithium# ping -A -c10 beryllium PING beryllium (172.31.1.2) 56(84) bytes of data. 64 bytes from beryllium (172.31.1.2): icmp_req=1 ttl=64 time=152 ms 64 bytes from beryllium (172.31.1.2): icmp_req=2 ttl=64 time=168 ms 64 bytes from beryllium (172.31.1.2): icmp_req=3 ttl=64 time=88.1 ms 64 bytes from beryllium (172.31.1.2): icmp_req=4 ttl=64 time=137 ms 64 bytes from beryllium (172.31.1.2): icmp_req=5 ttl=64 time=145 ms 64 bytes from beryllium (172.31.1.2): icmp_req=6 ttl=64 time=105 ms 64 bytes from beryllium (172.31.1.2): icmp_req=7 ttl=64 time=138 ms 64 bytes from beryllium (172.31.1.2): icmp_req=8 ttl=64 time=164 ms 64 bytes from beryllium (172.31.1.2): icmp_req=9 ttl=64 time=174 ms 64 bytes from beryllium (172.31.1.2): icmp_req=10 ttl=64 time=123 ms --- beryllium ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 1307ms rtt min/avg/max/mdev = 88.111/139.794/174.547/26.414 ms, pipe 2, ipg/ewma 145.244/143.915 ms lithium# ping -A -c1000 -q beryllium PING beryllium (172.31.1.2) 56(84) bytes of data. --- beryllium ping statistics --- 1000 packets transmitted, 991 received, 0% packet loss, time 102193ms rtt min/avg/max/mdev = 26.115/102.891/175.749/43.053 ms, pipe 3, ipg/ewma 102.295/92.228 ms
netem
alog netem
