Projet

Général

Profil

DPDK » Historique » Version 60

Laurent GUERBY, 28/12/2015 15:07

1 1 Laurent GUERBY
{{>toc}}
2 1 Laurent GUERBY
3 1 Laurent GUERBY
h1. DPDK
4 1 Laurent GUERBY
5 1 Laurent GUERBY
Traitement haute performance en logiciel, 10 Gbit/s en petit paquets line rate
6 1 Laurent GUERBY
7 1 Laurent GUERBY
h2. Liens
8 1 Laurent GUERBY
9 1 Laurent GUERBY
* http://dpdk.org
10 1 Laurent GUERBY
* [[AtelierPPS2012]]
11 16 Laurent GUERBY
* GIT http://chiliproject.tetaneutral.net/projects/git-tetaneutral-net/repository/dpdk
12 16 Laurent GUERBY
<pre>
13 16 Laurent GUERBY
$ git clone ssh://git@git.tetaneutral.net/dpdk.git
14 16 Laurent GUERBY
</pre>
15 16 Laurent GUERBY
16 17 Laurent GUERBY
* Nerim Antoine https://twitter.com/nerim_net/status/517592380841930752
17 17 Laurent GUERBY
** Du code maison fondé sur #DPDK en prod dans le réseau : première étape vers l'élimination des systèmes propriétaires ?
18 2 Laurent GUERBY
19 2 Laurent GUERBY
h2. Misc
20 2 Laurent GUERBY
21 45 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf
22 45 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/datasheets/82598-10-gbe-controller-datasheet.pdf
23 44 Laurent GUERBY
* MoonGen: A Scriptable High-Speed Packet Generator
24 44 Laurent GUERBY
** http://arxiv.org/pdf/1410.3322.pdf
25 44 Laurent GUERBY
** http://www.net.in.tum.de/fileadmin/bibtex/publications/papers/MoonGen_IMC2015.pdf
26 44 Laurent GUERBY
** https://github.com/emmericp/MoonGen
27 26 Laurent GUERBY
* https://www.kernel.org/doc/Documentation/networking/switchdev.txt
28 25 Laurent GUERBY
* https://www.usenix.org/conference/osdi14/technical-sessions/presentation/belay
29 25 Laurent GUERBY
** IX: A Protected Dataplane Operating System for High Throughput and Low Latency
30 25 Laurent GUERBY
** http://www.linuxfoundation.org/images/1/1c/Network_data_flow_through_kernel.png
31 25 Laurent GUERBY
** http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow
32 23 Laurent GUERBY
* https://blog.cloudflare.com/how-to-receive-a-million-packets/
33 23 Laurent GUERBY
** How to receive a million packets per second
34 24 Laurent GUERBY
* https://blog.cloudflare.com/how-to-achieve-low-latency/
35 21 Laurent GUERBY
* http://www.cs.cmu.edu/~dongz/papers/cuckooswitch.pdf
36 22 Laurent GUERBY
** We show that CUCKOOSWITCH can process 92.22 million minimum sized packets per second on a commodity server equipped with eight 10 Gbps Ethernet interfaces while maintaining a forwarding table of one billion forwarding entries.
37 18 Laurent GUERBY
* Improving Linux networking performance https://lwn.net/Articles/629155/
38 18 Laurent GUERBY
** https://www.youtube.com/watch?v=3XG9-X777Jo
39 19 Laurent GUERBY
** http://people.netfilter.org/hawk/presentations/LCA2015/net_stack_challenges_100G_LCA2015.pdf
40 20 Laurent GUERBY
* http://www.bigswitch.com/sites/default/files/presentations/onug-baremetal-2014-final.pdf
41 2 Laurent GUERBY
* Intel PCI id http://www.intel.com/support/network/sb/cs-012904.htm
42 3 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-products.html?iid=embed_portal+hdprod_ethernet#s1=Gigabit%20Ethernet&s2=all&s3=all
43 3 Laurent GUERBY
* Intel Pro/1000 PT vs CT http://hardforum.com/showthread.php?t=1386042
44 4 Laurent GUERBY
* http://images.ncix.com/forumimages/C9864B87-583B-10FB-B0DE80742781CE07.jpg
45 4 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-controllers.html#s1=all&s2=82572EI&s3=all
46 4 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-controllers.html#s1=all&s2=82574L&s3=all
47 5 Laurent GUERBY
* https://github.com/Pktgen/Pktgen-DPDK
48 6 Laurent GUERBY
* Evaluating the Suitability of Server Network Cards for Software Routers http://kfall.net/ucbpage/papers/snc.pdf
49 13 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/guides/intel-dpdk-sample-applications-user-guide.pdf
50 13 Laurent GUERBY
*  rte_eth_rx_burst
51 13 Laurent GUERBY
** http://dpdk.org/doc/api/rte__ethdev_8h.html#a5ac93faa0f3f69545ca121dcfd9433f5
52 13 Laurent GUERBY
** http://dpdk.info/ml/archives/dev/2014-January/001223.html
53 14 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/white-papers/aes-ipsec-performance-linux-paper.pdf
54 15 Laurent GUERBY
* http://networkbuilders.intel.com/docs/Network_Builders_RA_vBRAS_Final.pdf
55 15 Laurent GUERBY
56 15 Laurent GUERBY
* https://github.com/SnabbCo/snabbswitch
57 15 Laurent GUERBY
** https://github.com/SnabbCo/snabbswitch/blob/master/src/apps/intel/intel10g.lua
58 15 Laurent GUERBY
** http://blog.ipspace.net/2014/06/snabb-switch-and-nfv-on-openstack-in.html
59 15 Laurent GUERBY
** https://github.com/SnabbCo/snabbswitch/wiki/Snabb-Lab
60 15 Laurent GUERBY
** https://twitter.com/lukego/status/327714050219847680
61 7 Mehdi Abaakouk
62 7 Mehdi Abaakouk
h2. Note sileht dpdk:
63 7 Mehdi Abaakouk
64 7 Mehdi Abaakouk
Extract from: http://www.intel.com/content/dam/www/public/us/en/documents/guides/intel-dpdk-getting-started-guide.pdf
65 7 Mehdi Abaakouk
66 7 Mehdi Abaakouk
h3. configuration hugepages: 
67 7 Mehdi Abaakouk
68 7 Mehdi Abaakouk
* 2M (1024*2k): hugepages=1024 (at runtime: echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
69 7 Mehdi Abaakouk
* 4G (4x1G): default_hugepagesz=1G hugepagesz=1G hugepages=4 (only works at boot time via grub)
70 7 Mehdi Abaakouk
71 7 Mehdi Abaakouk
<pre>
72 7 Mehdi Abaakouk
mkdir /mnt/huge
73 7 Mehdi Abaakouk
mount -t hugetlbfs nodev /mnt/huge
74 7 Mehdi Abaakouk
</pre>
75 7 Mehdi Abaakouk
76 7 Mehdi Abaakouk
h3. Compile and load modules: 
77 7 Mehdi Abaakouk
78 7 Mehdi Abaakouk
_Note: source tools/setup.sh is a helper tools for this but works only with IGB driver not e1000e_
79 7 Mehdi Abaakouk
80 1 Laurent GUERBY
<pre>
81 8 Mehdi Abaakouk
# apt-get install make gcc linux-headers-amd64
82 8 Mehdi Abaakouk
# make config T=x86_64-default-linuxapp-gcc
83 7 Mehdi Abaakouk
# make T=x86_64-default-linuxapp-gcc
84 7 Mehdi Abaakouk
..
85 7 Mehdi Abaakouk
Build complete
86 7 Mehdi Abaakouk
87 7 Mehdi Abaakouk
# modprobe uio  (I think this is not useful)
88 7 Mehdi Abaakouk
# insmod build/kmod/rte_kni.ko (I think this is not useful)
89 7 Mehdi Abaakouk
# insmod build/kmod/igb_uio.ko (I think this is not useful)
90 7 Mehdi Abaakouk
# ./tools/pci_unbind.py --status
91 7 Mehdi Abaakouk
92 7 Mehdi Abaakouk
Network devices using IGB_UIO driver
93 7 Mehdi Abaakouk
====================================
94 7 Mehdi Abaakouk
<none>
95 7 Mehdi Abaakouk
96 7 Mehdi Abaakouk
Network devices using kernel driver
97 7 Mehdi Abaakouk
===================================
98 9 Mehdi Abaakouk
0000:00:19.0 '82579LM Gigabit Network Connection' if=eth1 drv=e1000e unused=<none>
99 9 Mehdi Abaakouk
0000:01:00.0 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth6 drv=ixgbe unused=igb_uio *Active*
100 9 Mehdi Abaakouk
0000:01:00.1 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth7 drv=ixgbe unused=igb_uio
101 9 Mehdi Abaakouk
0000:04:00.0 '82571EB Gigabit Ethernet Controller' if=eth2 drv=e1000e unused=igb_uio *Active*
102 9 Mehdi Abaakouk
0000:04:00.1 '82571EB Gigabit Ethernet Controller' if=eth3 drv=e1000e unused=igb_uio
103 9 Mehdi Abaakouk
0000:05:00.0 '82571EB Gigabit Ethernet Controller' if=eth4 drv=e1000e unused=igb_uio
104 9 Mehdi Abaakouk
0000:05:00.1 '82571EB Gigabit Ethernet Controller' if=eth5 drv=e1000e unused=igb_uio
105 9 Mehdi Abaakouk
0000:06:00.0 'RTL8111/8168B PCI Express Gigabit Ethernet controller' if=eth0 drv=r8169 unused=<none> *Active*
106 1 Laurent GUERBY
107 1 Laurent GUERBY
Other network devices
108 1 Laurent GUERBY
=====================
109 1 Laurent GUERBY
<none>
110 9 Mehdi Abaakouk
</pre>
111 7 Mehdi Abaakouk
112 9 Mehdi Abaakouk
test sur eth7:
113 9 Mehdi Abaakouk
114 9 Mehdi Abaakouk
<pre>
115 9 Mehdi Abaakouk
# ip link set eth7 down
116 9 Mehdi Abaakouk
# ./tools/pci_unbind.py --bind=igb_uio 0000:01:00.1
117 7 Mehdi Abaakouk
</pre>
118 7 Mehdi Abaakouk
119 7 Mehdi Abaakouk
h3. Prepare examples programs:
120 7 Mehdi Abaakouk
121 7 Mehdi Abaakouk
<pre>
122 7 Mehdi Abaakouk
# export RTE_SDK=/root/sileht/dpdk-1.6.0r1
123 7 Mehdi Abaakouk
# export RTE_TARGET=build
124 7 Mehdi Abaakouk
# cd /root/sileht/
125 7 Mehdi Abaakouk
# cp -r $RTE_SDK/examples/helloworld my_rte_app
126 7 Mehdi Abaakouk
# cd my_rte_app
127 7 Mehdi Abaakouk
# make
128 7 Mehdi Abaakouk
</pre>
129 7 Mehdi Abaakouk
130 7 Mehdi Abaakouk
131 7 Mehdi Abaakouk
h3. Tests
132 7 Mehdi Abaakouk
133 7 Mehdi Abaakouk
<pre>
134 1 Laurent GUERBY
</pre>
135 10 Mehdi Abaakouk
136 11 Mehdi Abaakouk
h2. Limitation SPF carte intel:
137 10 Mehdi Abaakouk
138 11 Mehdi Abaakouk
Une fois le module igb_uio changé sur les:
139 10 Mehdi Abaakouk
140 10 Mehdi Abaakouk
<pre>
141 10 Mehdi Abaakouk
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
142 10 Mehdi Abaakouk
</pre>
143 10 Mehdi Abaakouk
144 11 Mehdi Abaakouk
Un flags de l'eeprom change et le module ixgbe ne charge plus les ports de la carte qui n'ont pas des SFP+ intel avec le message d'erreur suivant:
145 10 Mehdi Abaakouk
146 10 Mehdi Abaakouk
<pre>
147 10 Mehdi Abaakouk
[  563.069683] ixgbe 0000:01:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
148 10 Mehdi Abaakouk
[  563.069786] ixgbe 0000:01:00.0: Reload the driver after installing a supported module.
149 10 Mehdi Abaakouk
</pre>
150 10 Mehdi Abaakouk
151 10 Mehdi Abaakouk
Pour contourner le problème:
152 10 Mehdi Abaakouk
153 10 Mehdi Abaakouk
<pre>
154 10 Mehdi Abaakouk
modprobe ixgbe allow_unsupported_sfp=1 # allow_any_sfp=1 suivant le kernel
155 10 Mehdi Abaakouk
</pre>
156 10 Mehdi Abaakouk
157 10 Mehdi Abaakouk
ou 
158 10 Mehdi Abaakouk
159 1 Laurent GUERBY
<pre>
160 1 Laurent GUERBY
echo "options ixgbe allow_unsupported_sfp=1" > /etc/modprobe.d/ixgbe_any_sfp.conf
161 1 Laurent GUERBY
update-initramfs -u -k all
162 11 Mehdi Abaakouk
</pre>
163 11 Mehdi Abaakouk
164 11 Mehdi Abaakouk
La carte fonctionnera à nouveau avec le message suivant:
165 11 Mehdi Abaakouk
166 11 Mehdi Abaakouk
<pre>
167 11 Mehdi Abaakouk
[  908.361767] ixgbe 0000:01:00.0 (unregistered net_device): WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics.  Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter.  Intel Corporation is not responsible for any harm caused by using untested modules.
168 10 Mehdi Abaakouk
</pre>
169 12 Laurent GUERBY
170 27 Laurent GUERBY
h2. Packet Journey
171 1 Laurent GUERBY
172 27 Laurent GUERBY
https://www.gandi.net/news/en/2015-12-16/6308-packet-journey_a_free_software_router_for_linux_based_on_dpdk/
173 27 Laurent GUERBY
https://github.com/gandi/packet-journey
174 27 Laurent GUERBY
175 27 Laurent GUERBY
h2. Atelier 2016
176 27 Laurent GUERBY
177 55 Laurent GUERBY
h3. Atelier Hardware
178 54 Laurent GUERBY
179 1 Laurent GUERBY
<pre>
180 28 Laurent GUERBY
core i7 2600 @ 3.40GHz
181 31 Laurent GUERBY
 fpu vme de pse tsc msr pae mce cx8 
182 31 Laurent GUERBY
 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 
183 31 Laurent GUERBY
 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl 
184 31 Laurent GUERBY
 xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 
185 31 Laurent GUERBY
 cx16 xtpr pdcm pcid sse4_1 sse4_2 
186 31 Laurent GUERBY
 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
187 28 Laurent GUERBY
16G RAM
188 28 Laurent GUERBY
SSD 250G
189 28 Laurent GUERBY
</pre>
190 28 Laurent GUERBY
191 28 Laurent GUERBY
<pre>
192 28 Laurent GUERBY
host eth DHCP          10G-a                                           10G-b
193 28 Laurent GUERBY
t1   54:04:a6:ef:a8:c7 00:1b:21:d9:28:55 fe80::21b:21ff:fed9:2855(t2b) 00:1b:21:d9:28:54 fe80::21b:21ff:fed9:2854(t3b)
194 28 Laurent GUERBY
t2   e0:69:95:2e:e6:a2 00:1b:21:d9:1a:e1 fe80::21b:21ff:fed9:1ae1(t3a) 00:1b:21:d9:1a:e0 fe80::21b:21ff:fed9:1ae0(t1a)
195 28 Laurent GUERBY
t3   bc:ae:c5:e1:b2:6d 00:1b:21:d9:df:3d fe80::21b:21ff:fed9:df3d(t2a) 00:1b:21:d9:df:3c fe80::21b:21ff:fed9:df3c(t1b)
196 56 Laurent GUERBY
197 56 Laurent GUERBY
1G e1000e
198 56 Laurent GUERBY
t1 00:15:17:a6:cb:19
199 56 Laurent GUERBY
t3 00:15:17:16:bd:de
200 27 Laurent GUERBY
</pre>
201 29 Laurent GUERBY
202 60 Laurent GUERBY
<pre>
203 60 Laurent GUERBY
h7   82599ES 8086:10fb 8086:7a11
204 60 Laurent GUERBY
g1   82599ES
205 60 Laurent GUERBY
g2   X520    8086:154d 8086:7b11
206 60 Laurent GUERBY
g3   82599ES
207 60 Laurent GUERBY
g4   X520 
208 60 Laurent GUERBY
g5   X520
209 60 Laurent GUERBY
g6   X520
210 60 Laurent GUERBY
n7   X520
211 60 Laurent GUERBY
g8   X520
212 60 Laurent GUERBY
g9   82598EB 8086:10f1 8086:a21f
213 60 Laurent GUERBY
g10  X520
214 60 Laurent GUERBY
g11  82598EB
215 60 Laurent GUERBY
g12  82598EB
216 60 Laurent GUERBY
stri X520
217 60 Laurent GUERBY
218 60 Laurent GUERBY
t1   82598EB 
219 60 Laurent GUERBY
t2   82598EB 
220 60 Laurent GUERBY
t3   82598EB 
221 60 Laurent GUERBY
222 60 Laurent GUERBY
stock 82598EB
223 60 Laurent GUERBY
stock 82598EB
224 60 Laurent GUERBY
stock X520
225 60 Laurent GUERBY
</pre>
226 60 Laurent GUERBY
227 55 Laurent GUERBY
h3. Atelier Liens
228 54 Laurent GUERBY
229 33 Laurent GUERBY
https://github.com/Gandi/packet-journey
230 30 Laurent GUERBY
http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html
231 40 Laurent GUERBY
http://dpdk.org/doc/quick-start
232 50 Laurent GUERBY
http://dpdk.org/browse/apps/pktgen-dpdk/tree/README.md
233 1 Laurent GUERBY
http://dpdk.org/doc/guides/testpmd_app_ug/index.html
234 47 Laurent GUERBY
http://dpdk.readthedocs.org/en/v2.2.0/
235 54 Laurent GUERBY
https://github.com/emmericp/MoonGen
236 29 Laurent GUERBY
http://www.slideshare.net/movilforum/7-hands-on
237 33 Laurent GUERBY
http://p4.org/wp-content/uploads/2015/11/Princeton-Demo-Poster.pdf
238 59 Laurent GUERBY
http://dpdk.org/ml/archives/users/
239 59 Laurent GUERBY
http://dpdk.org/ml/archives/dev/
240 59 Laurent GUERBY
http://openvswitch.org/support/ovscon2015/
241 59 Laurent GUERBY
242 33 Laurent GUERBY
243 54 Laurent GUERBY
h3. Atelier Issues
244 1 Laurent GUERBY
245 1 Laurent GUERBY
* Issues opened
246 33 Laurent GUERBY
** https://github.com/Gandi/packet-journey/issues/20 
247 1 Laurent GUERBY
** https://github.com/pktgen/Pktgen-DPDK/issues/52
248 1 Laurent GUERBY
** https://github.com/emmericp/MoonGen/issues/106
249 58 Laurent GUERBY
** http://dpdk.org/ml/archives/dev/2015-December/030798.html
250 1 Laurent GUERBY
251 54 Laurent GUERBY
h3. Atelier Build
252 54 Laurent GUERBY
253 1 Laurent GUERBY
<pre>
254 54 Laurent GUERBY
apt-get build-dep linux-image-3.16.0-30-generic 
255 57 Laurent GUERBY
apt-get install linux-tools-3.16.0-57-generic
256 54 Laurent GUERBY
apt-get install emacs24-nox libpcap-dev tcpdump iperf iperf3 screen rsync socat bzip2 xz-utils cmake python-sphinx
257 54 Laurent GUERBY
</pre>
258 54 Laurent GUERBY
259 54 Laurent GUERBY
<pre>
260 54 Laurent GUERBY
# Dans /etc/rc.local par exemple
261 1 Laurent GUERBY
modprobe uio_pci_generic
262 1 Laurent GUERBY
mkdir -p /mnt/huge
263 29 Laurent GUERBY
mount -t hugetlbfs nodev /mnt/huge
264 1 Laurent GUERBY
# si pas GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=4" :
265 1 Laurent GUERBY
echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
266 49 Laurent GUERBY
267 54 Laurent GUERBY
# Sources
268 54 Laurent GUERBY
269 29 Laurent GUERBY
git clone https://github.com/Gandi/packet-journey.git
270 1 Laurent GUERBY
git clone http://dpdk.org/git/apps/pktgen-dpdk
271 29 Laurent GUERBY
git clone http://dpdk.org/git/dpdk-stable
272 52 Laurent GUERBY
https://github.com/emmericp/MoonGen.git
273 1 Laurent GUERBY
mkdir ftp
274 1 Laurent GUERBY
cd ftp
275 52 Laurent GUERBY
wget http://dpdk.org/browse/dpdk/snapshot/dpdk-2.2.0.tar.gz
276 52 Laurent GUERBY
cd ..
277 1 Laurent GUERBY
tar xf ftp/dpdk-2.2.0.tar.gz
278 52 Laurent GUERBY
279 54 Laurent GUERBY
# Build
280 54 Laurent GUERBY
281 30 Laurent GUERBY
cd dpdk-2.2.0
282 32 Laurent GUERBY
make config T=x86_64-native-linuxapp-gcc
283 1 Laurent GUERBY
make install T=x86_64-native-linuxapp-gcc
284 32 Laurent GUERBY
export RTE_SDK=/root/dpdk-2.2.0
285 38 Laurent GUERBY
export RTE_TARGET=x86_64-native-linuxapp-gcc
286 38 Laurent GUERBY
./tools/dpdk_nic_bind.py -s
287 38 Laurent GUERBY
./tools/dpdk_nic_bind.py --bind=uio_pci_generic 0000:01:00.0
288 42 Laurent GUERBY
./tools/dpdk_nic_bind.py --bind=uio_pci_generic 0000:01:00.1
289 37 Laurent GUERBY
./x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -c 0x6 -n 2 -- -i
290 1 Laurent GUERBY
291 33 Laurent GUERBY
cd ../packet-journey
292 37 Laurent GUERBY
make
293 1 Laurent GUERBY
294 33 Laurent GUERBY
cd ../pktgen-dpdk
295 41 Laurent GUERBY
make
296 41 Laurent GUERBY
cd docs
297 41 Laurent GUERBY
make html
298 29 Laurent GUERBY
</pre>