DPDK » Historique » Version 112
« Précédent -
Version 112/113
(diff) -
Suivant » -
Version actuelle
Laurent GUERBY, 21/06/2018 09:47
- Contenu
- DPDK
DPDK¶
Traitement haute performance en logiciel, 10 Gbit/s en petit paquets line rate
Liens¶
- http://dpdk.org
- AtelierPPS2012
- GIT http://chiliproject.tetaneutral.net/projects/git-tetaneutral-net/repository/dpdk
$ git clone ssh://git@git.tetaneutral.net/dpdk.git
- Nerim Antoine https://twitter.com/nerim_net/status/517592380841930752
- Du code maison fondé sur #DPDK en prod dans le réseau : première étape vers l'élimination des systèmes propriétaires ?
- https://fd.io/wp-content/uploads/sites/34/2017/07/FDioVPPwhitepaperJuly2017.pdf
- FD.io - Vector Packet Processing One Terabit Software Router
Misc¶
- https://www.linuxjournal.com/content/userspace-networking-dpdk
- https://lwn.net/Articles/738912/ Reports from Netconf and Netdev 2017
- https://labs.ripe.net/Members/santiago_r_r_/a-software-based-approach-to-identify-heavy-hitters-in-dns-traffic
- https://medium.com/netflix-techblog/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99
- http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-May/017814.html
- processor C-state tuning for ceph
- https://www.solid-run.com/marvell-armada-family/armada-8040-community-board/
- http://dtrace.org/blogs/rm/2016/09/15/turtles-on-the-wire-understanding-how-the-os-uses-the-modern-nic/
- https://fd.io/
- http://developerblog.redhat.com/2015/06/02/can-you-run-intels-data-plane-development-kit-dpdk-in-a-docker-container-yep/
- DPDK like
- http://www.intel.fr/content/dam/www/public/us/en/documents/datasheets/xl710-10-40-controller-datasheet.pdf
- http://www.intel.fr/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf
- http://www.intel.fr/content/dam/www/public/us/en/documents/datasheets/82598-10-gbe-controller-datasheet.pdf
- MoonGen: A Scriptable High-Speed Packet Generator
- https://www.kernel.org/doc/Documentation/networking/switchdev.txt
- https://www.usenix.org/conference/osdi14/technical-sessions/presentation/belay
- IX: A Protected Dataplane Operating System for High Throughput and Low Latency
- http://www.linuxfoundation.org/images/1/1c/Network_data_flow_through_kernel.png
- http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow
- https://blog.cloudflare.com/how-to-receive-a-million-packets/
- How to receive a million packets per second
- https://blog.cloudflare.com/how-to-achieve-low-latency/
- http://www.cs.cmu.edu/~dongz/papers/cuckooswitch.pdf
- We show that CUCKOOSWITCH can process 92.22 million minimum sized packets per second on a commodity server equipped with eight 10 Gbps Ethernet interfaces while maintaining a forwarding table of one billion forwarding entries.
- Improving Linux networking performance https://lwn.net/Articles/629155/
- http://www.bigswitch.com/sites/default/files/presentations/onug-baremetal-2014-final.pdf
- Intel PCI id http://www.intel.com/support/network/sb/cs-012904.htm
- http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-products.html?iid=embed_portal+hdprod_ethernet#s1=Gigabit%20Ethernet&s2=all&s3=all
- Intel Pro/1000 PT vs CT http://hardforum.com/showthread.php?t=1386042
- http://images.ncix.com/forumimages/C9864B87-583B-10FB-B0DE80742781CE07.jpg
- http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-controllers.html#s1=all&s2=82572EI&s3=all
- http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-controllers.html#s1=all&s2=82574L&s3=all
- https://github.com/Pktgen/Pktgen-DPDK
- Evaluating the Suitability of Server Network Cards for Software Routers http://kfall.net/ucbpage/papers/snc.pdf
- http://www.intel.fr/content/dam/www/public/us/en/documents/guides/intel-dpdk-sample-applications-user-guide.pdf
- rte_eth_rx_burst
- http://www.intel.fr/content/dam/www/public/us/en/documents/white-papers/aes-ipsec-performance-linux-paper.pdf
- http://networkbuilders.intel.com/docs/Network_Builders_RA_vBRAS_Final.pdf
- https://reannz.co.nz/news/ruru-visualise-international-traffic-flows-real-time/
- https://github.com/SnabbCo/snabbswitch
- https://github.com/snabbco/snabb/
- https://lwn.net/Articles/713918/
- https://github.com/SnabbCo/snabbswitch/blob/master/src/apps/intel/intel10g.lua
- http://blog.ipspace.net/2014/06/snabb-switch-and-nfv-on-openstack-in.html
- https://github.com/SnabbCo/snabbswitch/wiki/Snabb-Lab
- https://twitter.com/lukego/status/327714050219847680
Note sileht dpdk:¶
Extract from: http://www.intel.com/content/dam/www/public/us/en/documents/guides/intel-dpdk-getting-started-guide.pdf
configuration hugepages:¶
- 2M (1024*2k): hugepages=1024 (at runtime: echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
- 4G (4x1G): default_hugepagesz=1G hugepagesz=1G hugepages=4 (only works at boot time via grub)
mkdir /mnt/huge mount -t hugetlbfs nodev /mnt/huge
Compile and load modules:¶
Note: source tools/setup.sh is a helper tools for this but works only with IGB driver not e1000e
# apt-get install make gcc linux-headers-amd64 # make config T=x86_64-default-linuxapp-gcc # make T=x86_64-default-linuxapp-gcc .. Build complete # modprobe uio (I think this is not useful) # insmod build/kmod/rte_kni.ko (I think this is not useful) # insmod build/kmod/igb_uio.ko (I think this is not useful) # ./tools/pci_unbind.py --status Network devices using IGB_UIO driver ==================================== <none> Network devices using kernel driver =================================== 0000:00:19.0 '82579LM Gigabit Network Connection' if=eth1 drv=e1000e unused=<none> 0000:01:00.0 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth6 drv=ixgbe unused=igb_uio *Active* 0000:01:00.1 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth7 drv=ixgbe unused=igb_uio 0000:04:00.0 '82571EB Gigabit Ethernet Controller' if=eth2 drv=e1000e unused=igb_uio *Active* 0000:04:00.1 '82571EB Gigabit Ethernet Controller' if=eth3 drv=e1000e unused=igb_uio 0000:05:00.0 '82571EB Gigabit Ethernet Controller' if=eth4 drv=e1000e unused=igb_uio 0000:05:00.1 '82571EB Gigabit Ethernet Controller' if=eth5 drv=e1000e unused=igb_uio 0000:06:00.0 'RTL8111/8168B PCI Express Gigabit Ethernet controller' if=eth0 drv=r8169 unused=<none> *Active* Other network devices ===================== <none>
test sur eth7:
# ip link set eth7 down # ./tools/pci_unbind.py --bind=igb_uio 0000:01:00.1
Prepare examples programs:¶
# export RTE_SDK=/root/sileht/dpdk-1.6.0r1 # export RTE_TARGET=build # cd /root/sileht/ # cp -r $RTE_SDK/examples/helloworld my_rte_app # cd my_rte_app # make
Tests¶
Limitation SFP carte intel:¶
Une fois le module igb_uio changé sur les:
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Un flags de l'eeprom change et le module ixgbe ne charge plus les ports de la carte qui n'ont pas des SFP+ intel avec le message d'erreur suivant:
[ 563.069683] ixgbe 0000:01:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected. [ 563.069786] ixgbe 0000:01:00.0: Reload the driver after installing a supported module.
Pour contourner le problème:
modprobe ixgbe allow_unsupported_sfp=1 # allow_any_sfp=1 suivant le kernel
ou
echo "options ixgbe allow_unsupported_sfp=1" > /etc/modprobe.d/ixgbe_any_sfp.conf update-initramfs -u -k all
La carte fonctionnera à nouveau avec le message suivant:
[ 908.361767] ixgbe 0000:01:00.0 (unregistered net_device): WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules.
Packet Journey¶
https://www.gandi.net/news/en/2015-12-16/6308-packet-journey_a_free_software_router_for_linux_based_on_dpdk/
https://github.com/gandi/packet-journey
Atelier 2016¶
Atelier Hardware¶
En salle milieu a Myrys.
http://pad.tetaneutral.net/p/dpdk
core i7 2600 @ 3.40GHz fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid 16G RAM SSD 250G
http://www.intel.fr/content/dam/www/public/us/en/documents/guides/gbe-server-selection-guide.pdf
h7 82599ES 8086:10fb 8086:7a11 g1 82599ES (VM maposmatic) g2 X520 8086:154d 8086:7b11 g3 82599ES g4 X520 g5 X520 g6 X520 n7 X520 (VM openstack) g8 X520 g9 82598EB 8086:10f1 8086:a21f g10 X520 g11 82598EB g12 82598EB stri X520 t1 82598EB t2 82598EB t3 82598EB stock 82598EB stock 82598EB stock X520
old obsolete:
host eth DHCP 10G-a 10G-b t1 54:04:a6:ef:a8:c7 00:1b:21:d9:28:55 fe80::21b:21ff:fed9:2855(t2b) 00:1b:21:d9:28:54 fe80::21b:21ff:fed9:2854(t3b) t2 e0:69:95:2e:e6:a2 00:1b:21:d9:1a:e1 fe80::21b:21ff:fed9:1ae1(t3a) 00:1b:21:d9:1a:e0 fe80::21b:21ff:fed9:1ae0(t1a) t3 bc:ae:c5:e1:b2:6d 00:1b:21:d9:df:3d fe80::21b:21ff:fed9:df3d(t2a) 00:1b:21:d9:df:3c fe80::21b:21ff:fed9:df3c(t1b) 1G e1000e t1 00:15:17:a6:cb:19 t3 00:15:17:16:bd:de
Atelier Liens¶
https://github.com/Gandi/packet-journey
http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html
http://dpdk.org/doc/quick-start
http://dpdk.org/browse/apps/pktgen-dpdk/tree/README.md
http://dpdk.org/doc/guides/testpmd_app_ug/index.html
http://dpdk.readthedocs.org/en/v2.2.0/
https://github.com/emmericp/MoonGen
http://www.slideshare.net/movilforum/7-hands-on
http://p4.org/wp-content/uploads/2015/11/Princeton-Demo-Poster.pdf
http://dpdk.org/ml/archives/users/
http://dpdk.org/ml/archives/dev/
http://openvswitch.org/support/ovscon2015/
https://events.ccc.de/congress/2015/Fahrplan/events/7523.html lecture: DDoS mitigation EPIC FAIL collection
https://github.com/aragua/libpacket
https://github.com/groud/dpdkcap
Atelier Issues¶
- Issues opened
- https://github.com/Gandi/packet-journey/issues/20
- https://github.com/pktgen/Pktgen-DPDK/issues/52
- https://github.com/emmericp/MoonGen/issues/106
- http://dpdk.org/ml/archives/dev/2015-December/030798.html
- http://dpdk.org/ml/archives/dev/2016-April/037331.html
- https://github.com/Gandi/packet-journey/pull/36
- https://github.com/Gandi/packet-journey/pull/38
Atelier Build¶
http://pad.tetaneutral.net/p/dpdk
# From debian jessie basic install # Packages apt-get -y build-dep linux-image-amd64 apt-get -y install linux-headers-$(uname -r) apt-get -y install git linux-tools emacs-nox libpcap-dev tcpdump iperf iperf3 screen rsync socat bzip2 xz-utils cmake python-sphinx tcpdump # huge pages, dans /etc/rc.local par exemple modprobe uio_pci_generic mkdir -p /mnt/huge mount -t hugetlbfs nodev /mnt/huge # or GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=4" : echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages # Sources mkdir ftp cd ftp wget http://dpdk.org/browse/dpdk/snapshot/dpdk-2.2.0.tar.gz wget http://dpdk.org/browse/apps/pktgen-dpdk/snapshot/pktgen-2.9.12.tar.gz cd .. git clone https://github.com/Gandi/packet-journey.git git clone http://dpdk.org/git/apps/pktgen-dpdk git clone http://dpdk.org/git/dpdk-stable git clone https://github.com/emmericp/MoonGen.git # Build DPDK tar xf ftp/dpdk-2.2.0.tar.gz cd dpdk-2.2.0 make config T=x86_64-native-linuxapp-gcc make install T=x86_64-native-linuxapp-gcc # about 2mn40s on core i7 2.6 GHz, 4m35s on Celeron 3205U 1.5 GHz export RTE_SDK=/root/dpdk-2.2.0 export RTE_TARGET=x86_64-native-linuxapp-gcc /root/dpdk-2.2.0/tools/dpdk_nic_bind.py -s /root/dpdk-2.2.0/tools/dpdk_nic_bind.py --bind=uio_pci_generic 0000:01:00.0 /root/dpdk-2.2.0/tools/dpdk_nic_bind.py --bind=uio_pci_generic 0000:01:00.1 /root/dpdk-2.2.0/x86_64-native-linuxapp-gcc/app/testpmd -c 7 -n 2 -- -i # http://www.intel.fr/content/dam/www/public/us/en/documents/guides/dpdk-testpmd-application-user-guide.pdf # Build pktgen (2.9.13+ doesn't compile) cd .. tar xf ftp/pktgen-2.9.12.tar.gz cd pktgen-2.9.12 make # about 24s ./app/app/x86_64-native-linuxapp-gcc/pktgen -c 7 -n 2 --proc-type auto -- -P -N -m '1.0,2.1' # save toto, edit toto and change both dst.mac start N xxxx and add start 0 and start 1 at the end of the file ./app/app/x86_64-native-linuxapp-gcc/pktgen -c 7 -n 2 --proc-type auto -- -P -N -m '1.0,2.1' -f toto # Build MoonGen cd .. cd MoonGen git submodule update --init ./build.sh # about 42 seconds ./build/MoonGen ./examples/benchmark/udp-throughput.lua 1:1 # TODO cd ../packet-journey make cd ../pktgen-dpdk make cd docs make html
Patch pour packet journey sur dpdk 16.11: https://github.com/Gandi/packet-journey/pull/36