Trying to communicate between two VM's.
Have tried with igb v4.3 and ixgbe 3.17 both having same weirdness.
Other party is not able to send out Unicast packets (uses PXE inside VM) - only multicast packets sent.
Saw, there is a setting, which disables VM-to-VM internal switch, which might help to investigate problem on transferring packets through the external network instead of switching them internally.
Have you any tips or recommendations, how to configure linux host ( kernel 3.8.0-21 ) side driver
to overcome this as I think need to be config issue ?
Thanx for posting to our site.
While you didn't provide much in the way of details, I can make one educated guess.
VF to VF traffic is only routed internal to the 82599 if the VF's are on the same PF. If they are on different PF's, then they must go out to the physical switch and come back in on the other PF to get routed to the other VF.
Try to ensure the VF's are on the same PF. If you are still having issues, then please provide some additional details and I will see what I can do.
This is just strange guess, but if I set mac's & vlan tags, does it hurt the VF's functionality as this might hint : http://firstname.lastname@example.org/msg02733.html Re: [E1000-devel] ixgbe: macvlan on PF/VF when SRIOV is enabled
That depends on where and how you create the VLAN tags.
You should set MAC and VLAN tags from your hypervisor, which in turn talks to the PF and it programs the MAC and or VLAN tags down in the hardware.
If you try to setup some sort of VLAN tag in the OS of your VM, it likely will not work because the underlying filtering in the NIC will see a VLAN that it doesn't recognize (because the hypervisor didn't program it) and throw it away.
Yes, know the nature of that.
Think the problem is not in VLAN's because the traffic goes on using multicast and other party using ixgbevf -driver. But when trying PXE using unicast, the TX packets stays in VF TX -queue (8 slots size) and after that's filled up, TX overflow message generated.
So tried using different versions of PXE under qemu and that combination caused problems.
Want to correct, what I said on prev. msg : "that combination caused problems" meaning "that was not working with any of the combinations".
Testing that the Unicast packets flown away from PXE/TX, but they seemed not to occur on real network.
So just to clarify - you are unable to PXE boot a VM that is assigned a VF - is that correct?
If that is the case, this is expected behavior. The network code in a VM PXE setup doesn't know how to talk to a VF, only an emulated NIC device. In this case you will need to have an emulated NIC assigned to the VM for PXE and then when the OS is up and running, it can use the VF with the VF driver.
Please clarify the desired usage and your setup if this is not correct.
You are right - that's the setup.
On that purpose, I have delivered ixgbevf.rom for the PXE as a NIC's rom-file, and that's working correctly in terms of detecting NIC, and handling it - but only multicast was working fine. Am debugging driver (and PXE side also) so might this give a light what's wrong on. Any other ideas, if ixgbe host driver itself needs some configs in case of L2 Unicast to work ?
Cool, I find this interesting.
You say you get multicast traffic? Are you sure you don't mean broadcast?
So by default the L2 Sorter in the PF will send broadcast traffic to all VF's. Unicast traffic will only be sent based upon MAC and or VLAN filters. Have you programmed the VF's MAC address down in the hardware? This is generally done in the hypervisor using iproute2.
On the place where originally Unicast packets sent as there is Dest-MAC programmed in nothing goes into receiver as well as I know.
This is hard to say 100% true, because part of traffic can't be seen if it gets switch using internal L2-switch, and 2ndly if packets were flown on localhost not reaching my monitoring point.
Need to say also, the DHCP & BOOTP were successfully before this point, but application itself originally tried Unicast.
So from the start :
1.) PXE started in VM2
- next commands given manually from iPXE shell
2.) "dhcp net2" --> ok, VM1 given ip-parameters
3.) "imgfetch --> ok, VM1 given app.packet
4.) "boot --> NOT ok with Unicast, OK using Multicast using test-version. boot tried to communicate using same net2 loading a lot of app.software
This is same symptom in using igb v4.3 and ixgbe v3.17 in VM2 and seems some how relate to PXE usage in the way I cant understood.
Now I try to move VM1 into other host-machine and reroute traffic so that my monitoring point gets all the traffic (tcpdump).
Also interesting question is if the Unicast TX-side in VM2 needs some driver-setting using ip/iproute2 to enable Unicast packets reach the VM1 destination..
Thank you for interest to this question.
I'm not sure I understood all of what you wrote, I'll have to think on it for a while.
As for your last question, a VM can't modify the sorter down in hardware except for it's own MAC or VLAN tag. If both VM's are on a VF then it should work.
However if one of the VM's is on an emulated NIC, then you will have a problem because the emulated path is done in software and not in hardware, so the L2 sorter won't know anything about the emulated MAC so it will send it out the PF and the switch of course will not turn it back around to the same port.
Somhow, I might been in wrong track on point Unicasting.
After more debugs on, might be related to place, when VF tries to change MTU size and that causes VF disabled.
Can you please give some info about JF's, cause somehow this is relating to MTU size and JF's ?
( cut from ixgbe_sriov.c )
* If the PF or VF are running w/ jumbo frames enabled
* we need to shut down the VF Rx path as we cannot
* support jumbo frames on legacy VFs
As normally with mtu 1500 in PF, VFs were disabled cause some of the VF's trying to set higher MTU's. So how take VF 1.1. in use, which might be the answer ?
I think we are getting closer!
The 82599 does not have the capability to set a MTU for a VF that is larger than that of the PF. So if you want to have Jumbo Frames in a VF, you must set the size of the PF MTU to be at least as large as the VF.
I also believe the latest and greatest source has some updates in this area.
Please keep us posted!
Already have ixgbe 3.17.3, so not finding newer versions related to this.
But if I try to set "Jumbo Frame" size someting to 3k, driver with additional debugs says :
[876218.685342] ixgbe 0000:04:00.0 eth10: NM: VF 12 NETIF_F_FCOE_MTU disabled, IXGBE_FCOE_JUMBO_FRAME_SIZE 3072
[876218.685989] ixgbe 0000:04:00.0 eth10: NM: pf_max_frame 3014 max_frame 1540 ETH_FRAME_LEN 1514 ETH_FCS_LEN 4
[876218.687618] ixgbe 0000:04:00.0 eth10: >>VF max_frame 1540 out of range
As I before said, driver reported that it can't run VF's with jumbo frames enabled.
Otherwise, if I left PF's MTU size unchanged getting next erroneus debugs :
[1113770.681110] ixgbe 0000:04:00.0 eth10: NM: VF 12 ixgbe_set_vf_lpe(): chip is 82599B -type and pf_max_frame 1514
[1113770.681790] ixgbe 0000:04:00.0 eth10: NM: VF 12 NETIF_F_FCOE_MTU disabled, IXGBE_FCOE_JUMBO_FRAME_SIZE 3072
[1113770.682458] ixgbe 0000:04:00.0 eth10: NM: pf_max_frame 1514 max_frame 1540 ETH_FRAME_LEN 1514 ETH_FCS_LEN 4
[1113770.684131] ixgbe 0000:04:00.0 eth10: >>VF max_frame 1540 out of range
So how to proceed ?
Am far away from original title about "L2 Unicast" - sorry.
Now I have more clear information as the VM's are communication through different host and I do have possibility to tcpdump the whole traffic.
The ixgbe 3.17.3 has a feature of which I don't understand :
1.) Let's say I left PF's MTU unchanged on driver load. My VM trys to set MTU size 2000 to VF's which means, RX path of that VF is immediately disabled.
2.) If I increase PF's MTU size let's say to 2048, comparison on ixgbe detects that and makes same behavior, as the comment in source says "If the PF or VF are running w/ jumbo frames enabled, we need to shut down the VF Rx path as we cannot support jumbo frames on legacy VFs"
The comment above says that we do have Legacy VF's on use. Don't know, where this can be changed ?
So in clear : How to enable VF to use frame size larger than default (successfully), because use of VLAN's expects that frame size increases at least of those few bytes and on top of that : VM wants to increase MTU size to 2000 ?
And : ixgbe 3.17.3, kernel 3.0.8-21
Yes, you are right - expecting that the mbox api is in version 1.1.
I hard-code it driver, and things start to work. Is it so that someone should call this API and change this to 1.1 in order to have jumbo's enabled ?
At least in my case no-one calls "ixgbe_negotiate_vf_api()". Can you please give some light to this ?
/* reset VF api back to unknown */
// adapter->vfinfo[vf].vf_api = ixgbe_mbox_api_10;
e_info(probe, "VF %d VF API 1.0 -> 1.1 TEST ONLY !!!!!!!!!!\n", vf);
adapter->vfinfo[vf].vf_api = ixgbe_mbox_api_11;
This thread seemed to "eaten" my reply, so trying again. Just set has hard-coded mbox_api 1.0 --> 1.1 (inside driver) and after that MTU settings were OK and communication started with variable packet-sizes. Don't know how this mbox_api thought to be controlled as no one calls "ixgbe_negotiate_vf_api()". ?
-- edit : delayed transfer of msgs