FPGAs for robotics vision camera with 10Ge interface (10 gigabit ethernet)

[Altera_Forum]

I need to design a high-performance camera for robotics vision applications, and I'm having massive difficulty figuring out how to select the best combination of FPGA and PHY (~10Gbps transceivers).

I assume the best existing interface is 10Ge (10 gigabit ethernet), but I'm open to better alternatives if they exist.

In a working system, the cables from 4 of these cameras will need to plug into one [or two] interface cards in a high-speed, many-core PC.

At absolute minimum the cables between camera and PC need to be 30 meters, but they should be 90 to 100 meters. In many cases the 4 cameras will be in fixed positions (the four corners of a room at ceiling level), but in many other cases the 4 cameras will be attached to the moving robot (robotic device), presumably with cables hanging down from above. Therefore, an interface and cable system that remains reliable even as the robotic devices and attached cables move is highly desirable.

In making our choices of components and interfaces, the price of the entire system must be minimized (not one specific component). The following is a list of components and interfaces under consideration (other components that are not variable (like image sensor, case, PCB, etc) are not shown):

- FPGA (cyclone, arria, etc)

- 10Ge PHY (manufacturer part number)

- FPGA <==> PHY interface (XGMII, XAUI, etc)

- cable interface (10GBASE-T, various fiber-optics choices)

- cable connectors (RJ45, SPF+, various fiber-optics choices, etc)

- PC interface/client PCB for 4 cameras (presumably PCIe 3.0, # of lanes, consider other options)

Remember, the system requires we connect 4 cameras to the PC. Assuming the 4 camera cables plug into one (?or two?) interface/client cards, we must pay close attention to at least three considerations: PC bus bandwidth, PCIe slots, PCIe lanes. Assuming we only plug 4 cameras into a single PC (rather than 8, which is preferred but perhaps not practical), we have a continuous, sustained flood of 40Gbps (8GBps) of data from the cameras that needs to pass across the PCIe bus. This will require at least 8 lanes of PCIe 3.0, but in practice, more likely 16 lanes (the same as top-end GPU cards). In theory this could be distributed across two PCIe 3.0 interface cards with 8 lanes each... assuming motherboard support. I have a feeling far more motherboards will offer two 16-lane PCIe 3.0 slots (to support two GPU cards) than one 16-lane PCIe slot (for GPU) and two 8-lane PCIe slots (for other stuff). But that's just a guess.

-----

Now I'll make a few comments based upon my preliminary research. First and foremost, I am MASSIVELY confused about 10Ge cable/connector/interface options. No matter how many times I read articles and documents, I still don't understand half of what I'm reading. Note that I already designed a vision camera with Cyclone3 FPGA and Marvell 88e1111 PHY (1Ge via RJ45), so it isn't quite as if I'm a completely blank slate on the general topic of ethernet!

Almost nobody talks about prices, so I'm not able to narrow down the research process to something finite (like a specific 10Ge cable/connector/interface type). One thing I may have recognized is... it seems like client cards for 10Ge may be significantly cheaper for certain optical formats than 10GBASE-T copper with familiar RJ45 connectors. I have not seen comments about whether 10GBASE-T with CAT7 and RJ45 will be more or less reliable than the various fiber-optics interfaces when the robots and cables are moving around. I have to assume all interfaces are reliable with fixed cables.

I'm also not sure what's the deal with FPGA prices! I looked at the price of an Arria V part on digikey, and the prices ranged from $2,000 to... the price of a brand new Mercedes! WTF is this? The cyclone3 part in my previous camera cost $14 in unit quantity!

I understand that some of the newer and fancier FPGAs have some spiffy stuff inside (like fast transceivers, microcontrollers, and so forth). But... give me a break! Something doesn't make sense! Obviously I'm missing something.

Which brings me to the FPGA <<===>> PHY interface. The reason the cyclone3 part could drive the 1Ge PHY in my previous camera was because the FPGA <<===>> interface was parallel (8-bits data-in DDR, 8-bits data-out DDR, plus a few control bits). 1000Mbps / 8 = 125MHz DDR (which is a bit like 64Mbps SDR). That's an I/O switching rate that inexpensive FPGAs like cyclone3 can handle.

In the case of 10Ge, the nominal interface is called XGMII, which has 32-bit data-out DDR, 32-bit data-in DDR, plus a few control bits. 10000/Mbps / 32 = 312.5MHz DDR. That's certainly faster than 125MHz DDR, but I would guess should be within the ability of many-year newer FPGAs. True or false? I have to admit, trying to figure out the answer to questions like this from the FPGA specs is like pulling hair (even more so than the cyclone3 process was).

However, there may be another problem. Do any quad (or dual/single) 10Ge PHYs have XGMII interfaces? My research finds lots of 10Ge PHYs that seem to have various 4-bit in and 4-bit out interfaces (XFI, XAUI, others), but I'm not sure I've found any that have XGMII interfaces. What's strange is, some seem to claim XGMII support one place, but when I get to the block diagrams, they aren't visible (or aren't clear). For example, if you go to the following webpage (http://www.marvell.com/transceivers/alaska-x-gbe) and look at the 88X3240P and 88X3140 (or 88X3120) the description includes XGMII and Cu (copper). Yet when you click on the "product brief" link next to those product descriptions and look at the block diagrams, well, it isn't clear whether those parts support XGMII or not! Unfortunately, I haven't been able to get hold of the detailed specifications with BGA pinouts, which would probably let me figure this out definitively. Even though I have an NDA with Marvell from years ago (for the 88e1111 part and others), it has been an endless runaround for months on these two chips (not sure why, but maybe because I look like "small fry", given I'm an independent design engineer, not some mega-corporation).

Anyway, the question of XGMII versus XAUI (or one of the other 1-bit or 4-bit interfaces) MAY BE HUGE when it comes to what FPGA I need to drive the PHY. After all, each receiver or transmitter of a 4-bit interface for 10Ge has to support something like 3.125Gbps (presumably over some sort of two-wire differential scheme like LVDS for each bit in each direction). Yes, the data rate is only 2.500Gbps, but with the overhead of 8b/10b and frame start/stop markers (or whatever), the signal rate has to be 3.125Gbps (or close to that, I believe).

This is where many of you in the forum may have experience that makes this question easy! What are the cheapest FPGAs that can handle the (presumably 312.5Mbps DDR) signal rates of the 32-bit XGMII interface, and what are the cheapest FPGAs that can handle the (presumably 3.125Gbps LVDS) signal rates of XAUI or one of the other FPGA <<===>> PHY interfaces?

Incidentally, I can place the FPGA BGA right smack up against the PHY BGA, so PCB trace-length should not be an issue (probably in the range of 5mm to 20mm trace length, with exact equal length for + and - of each signal in the case of LVDS). The PCB will be somewhere between 8-layer and 12-layer, so signal routing between FPGA <<===>> PHY will not be a major issue.

Another price issue. I don't know whether 10Ge PHYs for 10GBASE-T are significantly more or less expensive than for fiber optics interfaces. If they are, let me know.

Other issues that some of you might be able to address:

Price of 30~100 meter long cat6e or cat7 cables with RJ45 connectors (this I work with already).

Price of 30~100 meter long fiber optics cables with [whatever] connectors (here I know nothing).

The difference between the above two is one part of the cost comparison. It is also important to explain the differences on the PCB between these two. In the case of my 1Ge device, the 88e1111 PHY connected right into a cheap "RJ45 with magnetics" connector (with something like 8 resistors and 8 capacitors in this circuit). I don't know whether the 10Ge connectors are any different or more expensive. I have no idea what the fiber-optics connector for mounting on the camera PCB will cost, and what (if any) other electronics or discretes are required on the PCB to support them.

Assuming the fiber optics cables already have connectors on both ends, I guess the cost of those connectors are irrelevant (included in the cable costs). This is the same for cat6e and cat7 cables with their attached RJ45 connectors.

I'm not sure, but the attached might be the order to figure out the choices mentioned above.

[Altera_Forum]

This isn't a 10G Ethernet answer, but: 

 If you're building cameras, you should check out CoaXPress and think along the lines of Cyclone V GT transceivers.  If you're coming from Cyclone III you'll probably find that more inline with your cost expectations?

[Altera_Forum]

@ted: 

 Okay, so instead of one 10Ge cable on each camera, you propose two CXP-6 connectors and cables (which could be bundled together and look like one cable)? That would transmit about 20% more data than 10Ge, which is good.  Looks like the required cable (Gepco VHD1100) costs about $1/ft == $3.30/meter which is a bit high, but not out of the question. Of course double these prices for two cables, so 50 meters * 2 == $330 for the cable, not including connectors on both ends. Looks like we might be talking $500 for the cables. Ouch. Nonetheless, I'll have to factor that in against everything else.  The transmitter chip and receiver chip seem to cost $32 in lowish quantities (~25), so that's another $128 per camera (two transmitters and two receivers). Add to the $500 estimate for cables and we're up to $628 (very roughly).  Each of the CoaXPress chips (ECQO62R20 receiver and ECQO62T20 transmitter) is designed to receive or transmit one 6.25GHz differential signal at CML levels. Presumably you intend the 6.25GHz differential output of the ECQO62R20 receiver IC to flow into 2 pins of one cycloneV GT receiver (or transceiver), which presumably means those cycloneV GT recievers and transmitters can input and output 6.25GHz+ differential signals at CML levels. The datasheet talks about 8b/10b encoding NRZ encoded data and says the bitrate limit is 6.25Gbps, so I guess that means "no DDR tricks to effectively get 12.5Gbps". :-)  The chips are small and appear quite convenient to apply, and I assume 8b/10b is trivial to implement in the FPGA. I suppose the chips wouldn't know better if the FPGA implemented something like 128b/130b instead to squeeze more data through. WOOPS. As I read the spec sheet, I assume the actual DATA throughput is 80% of 6.25Gbps == 5.00Gbps given the 8b/10b encoding. So I guess two of these cables will not be sending any more actual data than one 10Ge cable. Oh well, that's tolerable.  I still don't have a good price on the 10Ge PHY chips, but one vague hint indicates they might cost twice as much as these CoaXPress chips. I definitely need better information on PHY prices.  The cycloneV GT series FPGAs appear to cost roughly $200, $350, $500 for the three sizes. That's a lot cheaper than some of the altera FPGAs I was looking at for this project, but lots more than the $14 cyclone3 FPGA in my 1Ge camera. However, if these prices are 3x to 10x more than we'd actually have to pay in modest volume (100s or 1,000s and possibly 10,000s), then we're in the ballpark for this project. Depending on image sensor, we may need a lot of GPIO. Some of the cheap Aptina sensors have a very modest number of data-out signals, but some of the better and faster sensors output all 8,10,12,16 bits per pixel for several pixels at a time (up to 64 currently). That's a boatload of input signals! On the positive side, the wider the interface, the slower the bitrate at the FPGA GPIO pins, and thus the FPGA should be able to support the data rate easily. The Aptina sensors have a funky serial interface... maybe someone already developed IP for that interface.  Obviously the key question for the cycloneV GT is whether they can handle 6.25Gbps at CML levels. A quick look at the overview indicates they may support 6.144Gbps transceivers, but doesn't say whether that's DDR or CML. I'll have to read a bit further to figure that out, I guess. Obviously it isn't a killer to slow down the interface from 6.25Gbps to 6.144Gbps if that's all there is to it.  I see the overview says the cycloneV GX chips support 3.125Gbps. My crappy memory says 3.125Gbps is precisely what is required for the XAUI interface to some 10Ge PHYs (an alternative to the wide-parallel XGMII interface which requires much slower 312.5Mbps via DDR (so sorta 158MHz in a manner of speaking). The cycloneV GX FPGAs are somewhat cheaper than cycloneV GT, but not as much as I expected. Actually, the cycloneV E FPGAs aren't much cheaper either! Hmmm. I'll have to create a price matrix for all three series to understand what's happening here. Something doesn't make sense.  I had assumed a 10Ge PHY with XGMII interface would be the best choice, because the FPGA would be lots cheaper (because XGMII doesn't require fast receivers or transmitters or anything else... pretty much just a plain old FPGA capable of 160MHz/320Mbps DDR on GPIO). But maybe that's not true. Maybe the cycloneV GX that hopefully/probably handles the 3.125Gbps XAUI interface is just as cheap, or nearly as cheap. It does seem much easier to find 10Ge PHYs with XAUI interfaces. Not sure they're any cheaper, though.  Please correct any mistakes I made above, and expand on anything I said. Thanks for the idea. It might end up being a winner. Not sure yet.

[Altera_Forum]

In all your calculations you've forgot the NRE-Costs. 10Ge-IP-Core costs already 15-20000$! 

How big can the camera be? I mean, I would do it in this way:  Sensor -> Cyclone GT/GX/SX (SoC) -> PCIe-Host inside of FPGA -> some intel x540 10Ge-Card -> ... -> HOST PC.   Then you have to deal only with Sensor-to-FPGA interface and PCIe Interface and driver framework (Linux driver on SoC). No 10G-Interface, no phys, no headaches, no IP-core-costs. PCIe is for "free".  So, go to ALTERA Cyclone 5/Arria 10 SoC devkit, plug in the Intel PCIe X540 network card, design a small adapter board for your Aptina sensor and there you go.  Anyway, its big -- hardware AND software/driver takes some one man year of development, maybe more with custom PCB.  p.s.: sorry for my bad english

[Altera_Forum]

Kest makes a good point about using COTS components; Pleora might be worth a call: 

http://www.pleora.com/our-products/embedded-hardware/iport-ntx-ten

[Altera_Forum]

TO_BE_DONE

[Altera_Forum]

--- Quote Start ---  I read your messages, but am still confused.  First of all, I'm not trying to make one experimental device. I'm trying to design a product. Yes, a fairly specialized low volume product (only hundreds or at most thousands per year), but a product nonetheless. I'm a self-employed design engineer, but I expect to spend $25K to $100K developing this product. If I must spend $20K for IP, than I must. But I hope to avoid that if possible.   --- Quote End ---    I meant not experimental device, but prototyping device. I'm also design engineer and from my point of view I would't start such a big project not before prototyping.    --- Quote Start ---   If I read correctly, you want me to put a 10Ge PCIe card designed for PCs inside my camera. Yikes! I looked at the PCIe cards you mentioned, and they're fairly huge. While the camera doesn't need to be super tiny, what you propose is much larger than my plans.   --- Quote End ---    I understand, sorry I mixed up 10Gbit camera with IP-Camera.  For further explanations I have to say, that I'm not a guru regards 1Gbit or 10Gbit    --- Quote Start ---   My 1Ge camera contains no ethernet MAC. The FPGA takes image data directly from the sensor, puts that into ethernet packets, sets the various ethernet packet fields, then outputs them into a 1Ge Marvell 88e1111 ethernet PHY, which outputs the signals to the RJ45 connector (with integrated "magnetics"). No need for ethernet MAC.    --- Quote End ---    Okay, I understand. I would start with this: https://www.altera.com/products/boards_and_kits/dev-kits/altera/arria-10-soc-development-kit.html  Also look at this: http://wl.altera.com/products/ip/iup/ethernet/m-alt-10gbase-r-pcs.html  IP-10GBASERPCS http://eu.mouser.com/productdetail/altera/ip-10gbaserpcs/?qs=%2fha2pyfadujzlubt178%252bxkkstpwuqcbgtkq5ykydy6xfuqf9ukf%252buw%3d%3d  Then you need some fancy SFP+ module.  Till here you don't need to make a PCB, you can make your setup with some generated image/video and stream it to other PC or second Dev-Kit with SFP+ module.    --- Quote Start ---   BTW, since the above scheme doesn't have any CPU/processor involved in the process, there are no "device drivers" to write. I guess that's one man year of software saved! :-)   --- Quote End ---    Good point! :-D    --- Quote Start ---   The big question is. Is it really easier and/or cheaper to add the eMAC chip so the FPGA i/o is PCIe instead of XAUI ????   --- Quote End ---    I have no idea    --- Quote Start ---   # 1: no need for IP (or a bunch of work) to implement XAUI. # 2: no need for high-speed transceivers for FPGA <<===>> PHY.  I worry very much about the cost of high-speed transceivers on the FPGA!   --- Quote End ---    Yes, probably you will need a costly high-speed FPGA, like Arria 10.  I sum up: ARRIA 10 $900 SFP+ Module $300 (just a placeholder) IP-Core $50 ($5000 / 100 pcs) PCB $500 (just a random number) Other components: $200  So, you are at $1950 per board -- without the sensor, without other things... So let it be $4000 per camera (with development). I don't think you can use some Cyclone V FPGA for $100.      --- Quote Start ---   Note that many of the image sensors have the HiSpi interface, which is an LVDS interface with 4 data lines and 1 clock line (albeit only one direction... from image sensor to FPGA).  What worries me is the number of high-speed transceivers I will need on the FPGA, because the FPGA cost rises very fast when you need more transceivers. A whole slew of GPIO signals is cheaper than more high-speed transceivers. That's one major reason I have been hoping I can find a 10Ge PHY that supports the XGMII interface. That's a boatload of GPIO signals (72 == 36 inputs and 36 outputs), but they only need to support 320Mbps DDR == 160MHz i/o (which I sure hope is supported by cycloneV). Am I right about that? Does cycloneV GPIO support 320Mbps DDR ???  One consideration here is this. If we need high-speed transceivers for the image sensor (5 for HiSpi) and high-speed transceivers for XAUI (not sure if 4 for each direction or 4 transceivers), the number of high-speed transceivers is huge. In fact, I'm not sure the cycloneV even has enough high-speed transceivers in the largest parts for this! I see they have 6 or 9 high-speed transceivers, but apparently sometimes (some interfaces) an extra one is consumed for some magic reason).  Do you know? What's you take on this issue?   --- Quote End ---    You can handle it with Arria 10 without a problem, but I don't think you can use a Cyclone 5.     --- Quote Start ---   Now that I've explained better, please update your advice. And thanks a lot, this brainstorming helps!  --- Quote End ---    Resolution of the sensor? 4K60? Can you use a SDI? 12G-SDI? Image -> FPGA -> Hi-Speed-link -> Cable-Driver -> BNC Cable -> Blackmagic DeckLink 4K PCIe card...

[Altera_Forum]

--- Quote Start ---   Unfortunately, I haven't been able to find a 10Ge PHY with XGMII yet.   --- Quote End ---    http://www.mouser.com/search/productdetail.aspx?r=0virtualkey0virtualkeyvsc8486ysn-04

[joe306]

Hello, I do have a question that you may be able to answer. Could I connect a ECQO62R20.3 to the CML inputs/outputs of a SFP+ Fiber Transceiver module?

Thank you,

Joe