Categories: linux

SRIOV troubleshooting with Linux

SRIOV boogaloo

Basics of SRIOV

SRIOV is a specific method to bind a virtual NIC to the physical NIC of the hypervisor directly. There is no bridge/veth pair or faking in place. The VM is talking directly with the metal.

Concepts and terminology

  • The PF is the “Physical Interface” - the NIC card itself.
  • The VF is the “Virtual Function” - the fake NIC that is bound to the VM.
  • You can create multiple VF that are all bound to the same PF.
  • Each of those VF can be bound to the same VM or multiple different VMs.
  • Card drivers/manufacturers have different limits on the number of VF per PF. There is a performance hit after a certain number - I’ve seen it work on X710 cards, X520 (82599) - both from Intel. Mellanox also seems to support it. There have a lot of documentation regarding all the different settings that can be tweaked.

Why?

  • It mostly comes down to a performance requirements.
  • It’s “simpler” to manage as there is nothing between you and the hardware.
  • Very common in NFV deployments where low latency, high throughput is required.
  • Extremely dependant on drivers, manufacturer. YMMV depending on the mix of both.
  • You can bond VF on two different SRIOV ports to allow HA over two different TORs.

Things you won’t be able to do.

  • The kernel loses all control over the VF.
  • You cannot tcpdump a VF.
  • You can tcpdump a PF (the physical port itself) but results will vary. With the X710 cards, you will only see inbound (TOR) traffic (maybe even only broadcast packets).

Settings to tweak

  • Trusted VF
    • Allow to VF to enter promiscuous mode. Allow the VF to run a packet capture. There is some security concern as when the VF is trusted, the VM with the VF can see the IN/OUT traffic accross all other VFs (who might not be owned by the same tenant). Does not matter too much if the compute is only use by a single tenant.
  • Link-state
    • Will change the behavior of the VF when the PF is physically disconnected.
    • Enabled –> The VF will always remain physically connected even if the PF is physically down (cable removed or TOR shutdown)
    • auto –> The VF will reflect the physical state of the PF. Important to configure in failover scenario where the VM expects the physical state of the port to change.
  • Spoofcheck
    • During the assignation of the VF to the instance, a randomly generated MAC address is assigned. This setting can allow the VM to fake the outgoing MAC address of the packets.

Troubleshooting commands

  • Show the VF configured on an SRIOV interface.
    • ip link show $SRIOV_PORT_NAME

** ADD OTHER COMMANDS HERE! **

Laurent Dumont
I route packets and juggle bytes