Writing a Poll Mode Driver (PMD) for DPDK

Writing a PMD for DPDK

As mentioned in the previous post in this series about DPDK, I have
shared how to prototype L2/L3 applications using DPDK library
without any dependency on Hardware (Physical/Emulated).

In this post, I share how to write a PMD (Poll Mode Driver) for
DPDK, off-course, it will not cover all the enhancements and features present in industry grade drivers, which you can explore in
the existing PMDs in the DPDK source. Advanced features include,
queuing algorithms for QoS, Offloading features for Protocol
offloading, Protocol Parallelization features.

Here, we see how to write a PMD for a Network Interface Card,
most commonly found on your Desktop/Laptop, Realtek NIC card. The Realtek NIC on the physical hardware is 8136however, there is also a qemu-kvm based emulated NIC which can be configured on the virt-manager “8139”. Another emulated NIC e1000 exists which also has DPDK support, which also helps in doing tests to compare working and non-working implementations. Hence, I chose 8139 device to write a PMD. There is currently no PMD for Realtek 8139 devices, so its a good exercise to explore how PMDs are written as well as device drivers in General. The realtek NIC support in qemu-kvm is open-source and freely available. Writing a PMD for it should be considered purely educational and gives no competitive advantage over any other entity. Few years back, I attended a workshop from sysplay.in, and although I am not a driver developer currently, this was inspired from that session. It is a must attend workshop series for anyone who is interested in System Software interaction with hardware.

Using Virtual machine also helps to setup test environment on the host. For this purpose, my test environment includes 2 virtual machines running Ubuntu 18.04. I have enabled 2 NICS ( A & B) per Virtual machine. The goal of this experiment is to run dpdk-pingpong app as client and server. I used 1 virtual machine as a pingpong server on NIC-B which was based on e1000 emulated NIC, while the pingpong client is based on rtl8139 device (again NIC-B).

1. What is a PMD (Poll Mode Driver):
 If you have gone through DPDK in general, the processing of IOs and Network packets, is being pushed to userspace. Whenever a device needs application to work dedicatedly towards processing IO or Network packets, you need to bypass the kernel storage/networking stack and directly access the hardware. The reasons to skip kernel is to avoid multiple copies of data before its processed, as well as, avoid multiple copies in the other direction as well.

2. How to access hardware directly in User space:
The linux kernel has features (kernel modules) to work with hardware directly. The Userspace-IO is a kernel module that exposes device registers to userspace applications through files. The user-space application then simply needs to open the binary files and access specific offsets of the device registers.

3. How DPDK uses PMDs:
DPDK has build options to compile your driver as a shared library or compile everything as a single combined library.
These options can be set/unset through the “.config” configuration file present in the build directory. Like the Linux kernel, DPDK provides software bus enumeration and device registration facility which your driver should use to register itself to the bus. All the bus enumeration and device registration happens either before your main() gets called, or through library provided function later called for init.

4. Testpmd to test your driver:
DPDK provides a testpmd application to test your driver. However, its not straightforward to integrate your driver with the application since, there are no generic function pointers provided to work with the driver functions, hence you separately compile your driver as a library and then compile the
testpmd alongwith the driver library to use the driver. You also need to write multiple #ifdefs and #if defined MACROs for your Vendor specific driver to call functions directly to setup the device. I instead used the dpdk-pingpong application which works in client and server mode. So I can send ping packets using the custom PMD that I have written to the server.

5. How to write the basic framework:
All DPDK compatible PMD drivers are present under the drivers/net hierarchy, so we will have to add a directory for our vendor specific driver here: drivers/net/realtek. We need to add the configuration options to COMPILE source under our driver directory in build/.config, CONFIG_…
In the driver directory, the Makefile should export headers which contain all functions that our testpmd would call, and finally in the testpmd make a provision to include our driver library before compiling.
DPDK framework provides following list of function pointers for
any PMD to implement.

As I said earlier, it is not necessary to implement all of these, and you can keep many of them as “nop”. The typical order in which the dev_ops are called is as below:

So, based on this sequence and carefully studying an existing Linux driver for the same device (or if you wish to directly right a PMD, I think you should still refer to an existing implementation for device programmability), I am able to port the 8139cp.c driver in linux-4.15. Essentially, the probe function calling the init of the driver, can be accommodated in the .dev_configure and .dev_start routines. Setting up the TX queues for establishing DMA addresses should be done in the .tx_queue_setup. The DMA APIs like dma_alloc_coherent() have counterparts available in the DPDK library:

We attach our eth_dev_ops to dpdk provided eth_dev structure dev_ops handle. Also the 2 functions required to Transmit and Receive packets are also assigned to the DPDK provided handles, “realtek_recv_pkts” and “realtek_xmit_pkts”. We store the Register Space Base address from BAR1 into our device specific private structure struct realtek_eth_hw. The BAR1 is the non-prefetchable Memory mapped IO space provided in the lspci -vv output:

To transmit the packets refer to the below datasheet section:

As soon as rte_mbuf packet is filled into the already established TX descriptor space, we can set the NPQ bit in the TPPoll Register. The device will take care of sending the packets for you. It will also set TxOK flag in the Interrupt Status register as stated below:

So based on the above description, your status should read 0x4 for TxOK.

Few tests which I ran to confirm the address is being accessed correctly:
(these are device specific, so I will share the relevant data sheet section for reference)
1. Test the Device reset functionality:

As per the datasheet, I must write a Reset command to the Command Register. Then the device does a soft reboot and Resets (=0) the value of the Command register. To test this poll the command register for few seconds and if the Command Register is not reset to 0 then mark timeout and return error.

2. Run pingpong application to test the TX functionality:
On the VM-A where we use e1000 NIC, will work as a pingpong server.
On the VM-B where we use rtl8139 NIC, will work as a pingpong client.

At the time of writing this blog, I have not written the RX functionailty hence using the rtl8139 side for sending Pings alone, pongs would be sent by server, but not reported. Follow the instructions mentioned in the dpdk-pingpong github repository to build and run the app as client or server. Use “-l 0,1" instead of “-l 1,2” in case you only configure 2 CPUs in virt-manager.

Graphviz for Network Visualization

Graphviz for Network Visualization Draw network diagrams from config files Graphviz package converts dot(.) file based descriptions into graphs. It is not restricted to UML or any specific type of graphs and can be used for automating flowchart and Network diagrams as well. See Wiki In this case, I am using the tool to convert the Network plan mentioned in the below post into a diagram for easy reference. Build a network emulator using Libvirt and KVM In this post, I demonstrate how to create a network emulation scenario using Libvirt, the Qemu/KVM hypervisor, and…www.brianlinkletter.com On Ubuntu, do "apt install graphviz" and "apt install xdot" Once package is installed. Run the below commands for generating PDF or PNG files based on your preference. dot -Tpdf graph1.dot -o graph1.pdf dot -Tpng graph1.dot -o graph1.png The language is pretty simple, unless you wish to do any fancy stuff, then it would need deeper study, but for network diagrams, understanding one .dot file as mentioned below would be sufficient. The resultant network plan looks like this: References: 1. https://www.graphviz.org/doc/info/shapes.html 2. https://renenyffenegger.ch/notes/tools/Graphviz/examples/index 3. https://medium.com/p/9f45693d69d8/edit

Coorg — Coffee, Homestays and much more

 

Coorg — Coffee, Homestays and much more

Among the hill stations of India, Coorg has a special significance, Army Generals — Kariappa, Thimaiaha. Air Marshal K C Nanda as well as Defense Minister V.K. Krishna Menon are among the well known natives of Coorg (or Kodagu as the natives would call it). I was in Mysore, last in 2012 for my training in Infosys and missed visiting Coorg before I moved to Chennai. This time around, with due diligence, to cater to my parents holiday request we planned a short three day trip.

Coorg is famous for its home-stays (which is a way to commercialize your private house on a beautiful estate of coffee plants). It gives an opportunity to a city dweller like me to enjoy nature while staying in its midst, clean environment, fresh air and homely food all at a single place. I belong from konkan in Maharashtra and hence I am well aware of all these benefits, hence, if I get an opportunity to stay in a cottage, I always prefer that over a resort. We booked our arrangements from https://www.coorghomestays.com/, which seemed to me as an association of estate owners registering there properties for the general public to use.

Wild woods cottage is owned by Mr. Mukul Apaiaha, a 2nd generation plantation owner, and his estate spans a humbling 40 acres. Like the Thimaiaha’s and Kariappa’s of Coorg, Mr Apaiaha also has a proud military history associated with his estate. His father was in the Indian Army and invested in the estates early on. There were 2 cottages on the property both fifty plus years old, and had the traditional monsoon house style of architecture. My father explained to me the terms, Khidki and Phadki which a Darwaja is divided, the more common among them being khidki (window) while phadki meant the lower section of a door. The wood would have the best variety available locally, his dining table chairs were super heavy which made the quality evident. There was a small Porch with a central table and chairs on the upper floor, much like an observation deck for residents like us. His cottage is managed by Shantu (called with a Bengali pronunciation), and Gopal. On the night we reached the cottage, shantu guided us from the Kemalekad Estate gates to there abode. He has been given a jeep to pick-up and drop tourists and for other purposes. We had a good dinner, simple food, and Rasgulla for desert. We were already getting a feel of the weather change from Bangalore to Coorg, also it had rained so it was quite cool. We good see Fire Flies in the dark of the jungle around.

That day in the morning, we had taken the Srirangapatna route from Bangalore. On the way, we visited, Sri Ranganath Swamy Temple, and few places connected with Tipu Sultan, like the Daulat Baug on the banks of Cauvery. Been completed my quota of Temples (which I have also posted in COLORS of SOUTHERN INDIA.)

I was more inclined towards the historic gardens of Tipu Sultan. I was reminiscing about my visit to Kashmir during my childhood where I had visited Shalimar and Nishad baug near the Dal Lake. But that architecture can be witnessed at the Vrindavan Gardens in Mysore. Here its more of a lawn and a garden, rather than the water fountains. Tipu sultans summer cottage is well maintained and since it was a working day, we could patiently view all the exhibits. There were huge murals depicting the Anglo-Mysore wars, and paintings by British painters depicting the times then, and forts built around Mysore. ( I have made a point to trace down all the forts built by Tipu Sultan in subsequent days).

We visited the Namdroling Monastery the largest teaching center of the Nyingma lineage of Tibetan Buddhism in the world. The entire town of Bylakuppe is established to refuge the many thousands of Tibetan monks and lamas who fled during the Chinese aggression. The monastery has multiple temples with huge statues of Gautama Buddha, and vivid colorful murals depicting his life at lumbini, where he was born, Bodhi tree where he attained enlightenment and further years of his life. Earlier I used to believe that Tibet had a single spiritual leadership, under the Dalai Lama, Tenzing Gyatso, the famous monk who we granted refuge at Dharmashala in Uttarakhand. Turns out there are four sects within Tibetan Buddhism. Nyingma sect or school being the oldest among it. The other three are Kagyu, Sakya and Gelug. In contrast to the other three main Tibetan schools, the Nyingma tradition has never been the dominant political power in Tibet. The Dalai Lamabelongs to the Gelug school and has been politically, more stronger in terms of its followers.

On the way, is a small catchment dam over Cauvery, Balmuri Falls. It’s a good weekend getaway stop for groups to enjoy a free water park experience. The next day was leisure time for us at the cottage, I explored the area around. Plenty of ponds and domesticated animals, ducks, cows, dogs. There is an internal trail around the entire forty acre estate and shantu promised to show us around on the third day before we checked out. We had breakfast and set out for remaining site seeing. We could witness the beauty of Coorg in daylight which we missed while arriving due to rains and by the time we reached it was already dark. Surrounded by coffee plantations and paddy fields, my father explained the resemblance to Tea plantations in Assam. Atop the hill will be the plantations and downhill paddy fields which helps in proper water management. We visited abbey falls, which seemed to be a perennial waterfall, but it would be definitely great to visit it post monsoon when it would be flowing at full capacity. Raja seat and Madikeri Fort were near Madikeri Town. The fort is now a converted court and government office but still has some essence of the British occupancy. Thalakaveri is a religious site where the Kaveri river originates. My parents especially liked the temple and kund, from where the river water flows downhill. We purchased spices from the local society shop and some ginger wine, Coorg also specializes in homemade wines, from ginger, gooseberry, pomegranate, beetle leaf and paddy. We had tea back at the cottage and relaxed listening to jungle sounds, birds calling and insects screeching.

Coorg, the “Land of generals” hasn’t had a peaceful history pre-independence. Before the British, the kodavas (natives of kodagu) had to fight back Tipu sultan. In a bid to overthrow Tipu sultan the Kodavas sided by the British which allowed them to access certain privileges till date (more about that can be read here:https://www.outlookindia.com/magazine/story/coorg- diary/298952

It is my reasoning, that Coorg enjoyed a special status in the Armed forces, as well as before, during the British rule, in spite of having a small area and weaker in resources as compared to the neighboring states like the Madras, Bombay, Travancore and Mysore presidencies. In recent times the number of candidates for the armed forces has reduced.

The next morning, as promised, shantu took us for a hike around the estate. There was plenty more to see apart from the area around the cottage. He indicated to us the path of elephants passing the estate when in dire need for water. The elephants need to be driven off by making drum sounds. The trail went up hill and all I could see were coffee plants. This was my first experience of a plantation, my virtual experience was with Django and the Doctor in Quentin Tarantino’s movie. Shantu explained the different coffee varieties Arabica and Robusta. He shoowed us how the coffee plant is bent down after a certain height so it does not grow upwards but expands horizontally. This helps in better yield as well as easy plucking of the coffee beans. He also provided some manure rich organic compost as a complimentary gift while returning.

There were still places we didn’t touch like the dubare elephant camp, and intentionally wanted to make the trip less hectic. Would mark these places for a much impending visit soon. Till then cheers on the Ginger wine…