We've got a couple of new Supermicro 6016T-6RF+ boxes with Ubuntu 10.04 2.6.32-24-server kernels, and when connected through a dumb switch everything is happy between them and to other systems. But when connected through a 3com 2948-SFP things get strange. The two new systems have full connectivity to each other through the switch, but cannot ping or get a full arp on anything else on its LAN (IPs show up in the arp table, but not the MAC addresses). Likewise other systems on the LAN can't see the new systems (and IPs show up in their arp tables, but not MACs). The new systems can see each other fine over a crossover cable, too.
Flusing the 3com's cache doesn't help. Everthing shows up as gigabit and full duplex on both the local systems and the Supermicros. Changing the settings on the local systems doesn't help - going to half duplex or less bandwidth. Booting to System Rescue Disk, Gentoo-based with a 126.96.36.199 kernel doesn't change it. It makes no difference which port on the NICs is used. So far I'm not finding other reports of this. What can it be?
Edits: Everything shows up as gigabit and full duplex on both the switch and the Supermicros. Flushing the 3com's cache...
Update: It's not the 3com. We've tried it with another managed 3com model, and with an unmanaged Netgear, and it's failing the same way - that is, these systems can see each other, but not other stuff on the LAN. We've done specific IP assignments, and tried DHCP - they can't connect to the DHCP server.
Now, these systems were working to other systems across an unmanaged Belkin switch. And they can talk to each other over anything. But they can't talk to other stuff over anything except when the Belkin was working (which isn't here to retest with). We've also of course swapped cables, swapped the use of the 2 NIC ports, tried it with the crossover between the systems disconnected - to see of that being there was confusing things for the other port. Connectivity stays broken.
Update 2: Just spoke with Supermicro's SuperServer support line. They have no idea what's wrong.