SRIOV troubleshooting with Linux
Basics of SRIOV
SRIOV is a specific method to bind a virtual NIC to the physical NIC of the hypervisor directly. There is no bridge/veth pair or faking in place. The VM is talking directly with the metal.
Concepts and terminology
- The PF is the “Physical Interface” - the NIC card itself.
- The VF is the “Virtual Function” - the fake NIC that is bound to the VM.
- You can create multiple VF that are all bound to the same PF.
- Each of those VF can be bound to the same VM or multiple different VMs.
- Card drivers/manufacturers have different limits on the number of VF per PF. There is a performance hit after a certain number - I’ve seen it work on X710 cards, X520 (82599) - both from Intel. Mellanox also seems to support it. There have a lot of documentation regarding all the different settings that can be tweaked.
- It mostly comes down to a performance requirements.
- It’s “simpler” to manage as there is nothing between you and the hardware.
- Very common in NFV deployments where low latency, high throughput is required.
- Extremely dependant on drivers, manufacturer. YMMV depending on the mix of both.
- You can bond VF on two different SRIOV ports to allow HA over two different TORs.
Things you won’t be able to do.
- The kernel loses all control over the VF.
- You cannot
- You can
tcpdumpa PF (the physical port itself) but results will vary. With the X710 cards, you will only see inbound (TOR) traffic (maybe even only broadcast packets).
Settings to tweak
- Trusted VF
- Allow to VF to enter promiscuous mode. Allow the VF to run a packet capture. There is some security concern as when the VF is trusted, the VM with the VF can see the IN/OUT traffic accross all other VFs (who might not be owned by the same tenant). Does not matter too much if the compute is only use by a single tenant.
- Will change the behavior of the VF when the PF is physically disconnected.
- Enabled –> The VF will always remain physically connected even if the PF is physically down (cable removed or TOR shutdown)
- auto –> The VF will reflect the physical state of the PF. Important to configure in failover scenario where the VM expects the physical state of the port to change.
- During the assignation of the VF to the instance, a randomly generated MAC address is assigned. This setting can allow the VM to fake the outgoing MAC address of the packets.
- Show the VF configured on an SRIOV interface.
ip link show $SRIOV_PORT_NAME
** ADD OTHER COMMANDS HERE! **