Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4811 Discussions

ixgbe LLIpush does not seem to work correctly

JSeba1
Beginner
2,297 Views

My understanding of the LLI Push parameter for ixgbe is that it will cause a Low Latency Interrupt (bypassing the interrupt coalescing setting) when a TCP packet with the PUSH flag set is received. However, it looks like it just issues an LLI no matter what.

If I load ixgbe with LLIPush=0 and ping flood the system with the default value for interrupt coalescing (rx-usecs 1), I get the following RTT:

round-trip min/avg/max/stddev = 0.043/0.054/0.167/0.004 ms

If I crank up rx-usecs to 1000 (1 millisecond), I get exactly what I expect:

round-trip min/avg/max/stddev = 0.045/1.003/1.432/0.021 ms

eg. average RTT increases by about 1 millisecond due to the interrupt coalescing delay. If I run the same test with LLIPush=1, I get this;

rx-usecs 1

round-trip min/avg/max/stddev = 0.040/0.050/0.137/0.004 ms

rx-usecs 1000

round-trip min/avg/max/stddev = 0.040/0.050/0.150/0.004 ms

The interrupt coalescing setting has no effect. I would expect to see this if the TCP push was set - but in this case, I am just doing a simple ping, so the packets aren't even TCP, much less have the TCP push flag set.

I experimented with changing the value written to IXGBE_FTQF(0), since it looked incorrect to me. I changed it from (IXGBE_FTQF_5TUPLE_MASK_MASK << IXGBE_FTQF_5TUPLE_MASK_SHIFT) to (IXGBE_FTQF_PROTOCOL_COMP_MASK << IXGBE_FTQF_5TUPLE_MASK_SHIFT) and this seems to prevent non-TCP packet from issuing an LLI. However I used hping to verify that any TCP packet, regardless of flags, seems to trigger an LLI. I also tried only setting only the SIZE_BP and CTRL_PSH bits in L34T_IMIR with the same results.

Note that all of my testing was done with an 82599, but the datasheet does not document the L34_TIMIR bit for LLI on Push, contradicting what is used by the ixgbe driver.

0 Kudos
11 Replies
st4
New Contributor III
1,081 Views

Hi JasonSeba,

We will check on this.

rgds,

wb

0 Kudos
st4
New Contributor III
1,081 Views

Hi JasonSeba,

Not sure if you have gone thru our README docu. Can you check if you are using MSI or MSIX?

http://downloadmirror.intel.com/23042/eng/README.txt http://downloadmirror.intel.com/23042/eng/README.txt

LLI (Low Latency Interrupts) ----------------------------

LLI allows for immediate generation of an interrupt upon processing receive packets that match certain criteria as set by the parameters described below. LLI parameters are not enabled when Legacy interrupts are used. You must be using MSI or MSI-X (see cat /proc/interrupts) to successfully use LLI.

Hope this helps.

rgds,

wb

0 Kudos
JSeba1
Beginner
1,081 Views

Yes, I am using MSI. But its not really relevant anyway - the problem appears to be that LLIs are being generated when they are not supposed to be, not that LLIs are not occurring.

Incidentally, I don't see anything in the data sheet indicating that MSI is required to support LLIs. This is potentially a problem for the application I am working on. Can you point to any documentation supporting this?

Thanks,

Jason

0 Kudos
st4
New Contributor III
1,081 Views

Hi Jason,

Thank you for the update. Let me further check on this.

With regards to using MSI for LLI. It is stated in the README below

http://downloadmirror.intel.com/23042/eng/README.txt http://downloadmirror.intel.com/23042/eng/README.txt

LLI (Low Latency Interrupts) ----------------------------

LLI allows for immediate generation of an interrupt upon processing receive packets that match certain criteria as set by the parameters described below. LLI parameters are not enabled when Legacy interrupts are used. You must be using MSI or MSI-X (see cat /proc/interrupts) to successfully use LLI.

rgds,

wb

0 Kudos
st4
New Contributor III
1,081 Views

Hi JasonSeba,

Can you help us by providing the value you set in the following LLI setting?

LLIPort use (range: 0 - 65535) = ?

LLIPush: Valid Range: 0-1 = ?

LLISize: Valid Range: 0-1500 = ?

LLIEType Valid Range: 0-x8fff = ?

LLIVLANP Valid Range: 0-7 = ?

Thanks,

wb

0 Kudos
JSeba1
Beginner
1,081 Views

LLIPush=1, all other settings are left at default and I have not tested them.

0 Kudos
st4
New Contributor III
1,081 Views

Hi JasonSeba,

Thank you for the info.will check on this.

rgds,

wb

0 Kudos
st4
New Contributor III
1,081 Views

Hi JasonSeba,

I hope you don't mind provide additional info below for further checking:

1)ethtool logs

2)Switch configuration

3)Are you invoking the LLIPUSH via a config file? Please provide

us the config file if there is

4) Is the reverse side scaling (RSS) paramter enabled or disabled?

Thanks,

wb

0 Kudos
JSeba1
Beginner
1,081 Views

1) I'm not sure what you mean by "ethtool logs", could you elaborate on this please?

Here is the output from ethtool -i p2p4 (edit: original info was from wrong driver)

driver: ixgbe

version: 4.0.3

firmware-version: 0xd87c0000

bus-info: 0000:05:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

2) I am not using a switch. I am direct connecting two 82599-based NICs

3) There is no config file, I am loading ixgbe manually via "modprobe ixgbe LLIPush=1,1,1,1,1,1"

4) The RSS parameter is at default value, which in this case is 0

0 Kudos
st4
New Contributor III
1,081 Views

Hi JasonSeba,

Thank you for the info. I will check from the info you provided.

rgds,

wb

0 Kudos
st4
New Contributor III
1,081 Views

Hi JasonSeba,

Good day. Further checking, you may contact e-1000-devel community for support. Here is the link for the access

https://lists.sourceforge.net/lists/listinfo/e1000-devel https://lists.sourceforge.net/lists/listinfo/e1000-devel

Hope this helps.

rgds,

wb

0 Kudos
Reply