Starting out, with 2 to 12 Phi's

[ccnyc]

So I bought 2 of the 31S1P's (not yet in use) and am contemplating getting another 10 for a fluid dynamics simulation. I'm trying to figure out how to proceed. I apologize if some of these are stupid questions. I think they're reasonable though as my setup is a bit unique: luckily I have a few 3d printers, in case I need to print brackets or ducts, and a 3600 CFM fan with 15" diameter. Some questions:

• If a board supports the Phi, and if the board and processors have an adequate number of PCIe x16/x8 slots and PCIe lanes, is there anything else I need to check to ensure the board will support 2 of them? How about n of them?
• What motherboard would you recommend (or, even if not recommended, you know will work) for running 2 31S1P's? How about 6 31S1P's? (I am sure I'm not going to be able to fit all 12 on a single 20A circuit in my home, even if there exists a single mobo that will accept them).
• It looks like the card occupies 2 slots, or perhaps more when cooling is considered. Is this true?
• I assume my fan is probably large enough to handle even all 12 31S1P's and certainly 2 or 6 of them if properly placed and ducted. Is this reasonable? It clearly meets the CFM/pressure requirements, but it's a bit different on a large scale -- for example, the center of the fan has little airflow, like most fans, but now the center is big enough that could be a problem. I suppose I could do a fluid dynamics simulation just for this ...
• How do I obtain the compiler software? It's a non-commercial project at the moment, although it might become commercial at a later date (in which case I would be able to order commercial licenses, probably many of them, and many more Phi's). It has its roots in the academic world with a class I took in grad school, but I'm doing this on my own at this point.
• Lastly, although I'm currently running code I've written myself, I wonder: is there any open source CFD software that has been optimized for the Xeon Phi? I'm also interested in commercial software whether it works on the Phi, although that stuff is expensive for an individual so I don't expect to be able to use it at the moment.

Thanks for your help!

[Kevin_D_Intel]

I cannot help the majority of those items but with regard to the compiler software, we do not currently offer a non-commercial license but you can obtain a 30-day free evaluation for the Intel® Parallel Studio XE 2015 here: https://software.intel.com/en-us/intel-parallel-studio-xe

The MPSS software needed is available from the article Intel® Manycore Platform Software Stack (MPSS).

Good luck with your project!

[Robert_F_2]

I know very little but I do know the Z9PE-D8 WS and Z10PE-D8 WS have a tremendous number of x16 slots and are stated by Asus to be compatible with the Phi's. Dual E5 2600's spec'ed for the card on PCI 2.0 16.  I don't know if that means we can serve up data to the cards as quickly with pci 3.0 8 in a 16 bit slot.    With the dual e5's there are a total of 40 pci 3.0 lanes per chip.   That seems to leave room for somewhere between 4 cards (if they need all 16 lanes) or 8 cards if 8 pci 3 lanes per slot are can service the card.   Either way it's going to be a 2011 C602 or 612 motherboard with 2 xeon 2600 E5's.

I also have 2 of them sitting on my desktop waiting for a computer to put them in.   My purposes however are entirely as a development learning platform.   I'd like to pick up a couple more so that I can establish more than one compute node with more than one coprocessor board and put a few gpgpu's beside them to see how much time I can waste in bandwidth and latency tryiing to put the right algorithms on the appropriate cards.

At 12 mic's and associated hardware you aren't playing in the same zip code as me so you may 

[Sunny_G_Intel]

With respect to your last question about open source CFD code optimized for Intel® Xeon Phi™ coprocessors: 

• LBS3D  
• OpenLB

You can also refer to Intel® Xeon Phi™ Coprocessor – Applications and Solutions Catalog for updated list of open source and industry software optimized for Intel® Xeon Phi™ Coprocessors

Thanks

[ccnyc]

Thanks Kevin, Robert, and Sunny!

• $700/license is totally doable for someone thinking about purchasing 10 Phi's. I was afraid it was much higher.
• Those ASUS boards look good. With 40 lanes that should be sufficient for 5 Phi's for a single processor if, as I have heard, it doesn't really matter whether it's x8 or x16 as long as your application mostly fits in the Phi's RAM. So physical constraints may be the limit, because with 7 PCIe slots each they might be able to handle 4 Phi's (taking up 2 slots each) if the cooling fan is sufficient, or 3 (with 1 slot in between each Phi) if they need more room. (I assume given the electrical and timing constraints there's no such thing as a PCIe slot extender, but it is 2015, so maybe I'm wrong about that.) I can populate them with a single E5-2603, I assume, leaving plenty of headroom for upgrades.
• CFD software: that looks excellent! Until now I've been using a completely custom version, my own code in C++ which follows the general path of the "12 steps to CFD" Python tutorial on the web. But with the Phi I could go to 3 dimensions, which for technical reasons will allow me to use general purpose CFD code rather than writing it from scratch. Nice!

This will be fun. I'll probably order one of the ASUS MB's, a $200 E6-2603, and associated hardware soon, to test on the 2 Phi's I have, and if that works I hope to be able to order 10 more. :)

I think you all have already answered all my questions, except that some uncertainty remains in the compatibility with n cards, and the fan question is probably just something that must be tested to find out. It's certainly enough to get me started. Thanks!

[jimdempseyatthecove]

ccync,

The motherboards listed in post #3 slot-wise, in theory could support 4 Xeon Phi. However, there may be component placement and/or slot placement. While I do not have one of those boards, I do have an earlier P9X79 WS, a perfectly fine board, which I am very satisfied with. I have two Xeon Phi 5110P's installed. From the picture of the Z10PE-D8 WS and looking at the user manual, the bottom slot is one of the x16 slots, and also a slot you would have to use if you attempted to insert 4 Xeon Phi cards. Using a double wide card in this slot may be problematic. I am unable to install a double wide video card in the bottom slot of my system. This may or may not be an issue with these other boards.

What will be an issue is cooling. I chose to cool by placing the fans inside my case. The cooling of these cards is a tricky business. If you select a standard workstation (or gaming) tower you would be hard pressed to locate the four fans and duct work inside the case. There are other threads on this forum illustrating cooling solutions for dual 31S1P's (including my pictures). You should take the time to look at these to give you an idea as to what you are up against.

Lucky for you, you have a 3D printer. The strategy I suggest for you is (assuming your motherboard and case will permit 4 cards), is to select a case without and internal drive bay that usually sits in the lower front of the case. You may find, with staggered placement you can print two dual channel manifolds, one for the top two Xeon Phi's, and one for the bottom two Xeon Phi's. You might have to stagger the fan locations such that two can fit in the lower part of the front drive bay. And do a similar thing with the lower two Phi's in the area where the internal drive bays used to be located.

Designing the print file is an entirely different issue.

Oh, save some money for some noise cancellation headphones.

Show us your setup when you have a solution.

Jim Dempsey

[ccnyc]

Jim,

Thanks! That's exactly what I was looking to find out regarding motherboard support. In the worst case, I could try to carefully attack the case with a dremel if the motherboard is ok with a double wide card but the case isn't. 3 cards in one system, however, isn't bad either.

You're right about the design work being the biggest effort for the 3d printer. For that reason four identical ducts, or two dual-channel manifolds as you suggest could be the right approach. I checked on thingiverse and youmagine (3d printing sites) and didn't find anything at the time related to the Intel Phi, so perhaps it'll be worth posting there too.

I forgot to mention I just happen to have an XPower 1/3 hp X-41ATR industrial fan, which is 3600 CFM, far more than required. I think it's normally used for drying paint rather than cooling microprocessors. I use it for a wind tunnel. Obviously that's not going to fit inside the case, but I'm hoping to have a case with an open side.  Or a single duct: I think my ducts might look a bit different with this fan ... an alternative might just be to use 6 mil thick plastic sheeting, non-rigid, to ensure that enough air reaches the case. I'm curious about the pictures of other solutions and will check those out. I'm not sure my fan will scale to 12 Xeon Phi's ... the CFM seems to be enough at the highest setting but the ducting there would be rather complicated. Whatever I find for that, you're right, it will be loud!

I'll be glad to post the setup when it's operating. I had a bit of a delay to this point ... I'll be ordering the first motherboard, power supply, case, and processors/cooling shortly, and if that goes well the plan is for the next ten Phi's after that.

[McCalpinJohn]

I don't have any specific knowledge to add to this discussion, but I did want to say "Bravo!" and applaud ccnyc's enthusiasm and creativity!

Of course one should not be surprised that a person who wants to do CFD would have some ideas about how to set up airflow for cooling a bunch of Xeon Phis, but not that many people seem to be willing to roll up their sleeves and start fabricating.   Make mistakes!  Get Dirty!   But remember to set up lots of temperature monitoring facilities to power everything down automagically if you are not in the room when something goes wrong!

I know that the mainstream Xeon processors (especially the Xeon E5 v3's) have very effective protection against thermal problems, but I don't know if the Xeon Phis are in the same league in that regard.   We have one Xeon E5 v3 system with a defective thermal solution -- idling at 45-50C and only able to dissipate ~50 watts before hitting its thermal throttling temperature.  At this point it drops the frequency well below the nominal minimum AVX frequency of 2.2 GHz (to about 1.7 GHz when running LINPACK), but continues to fiddle with the frequency to keep it as high as possible without exceeding the 89 C throtting temperature.

[ccnyc]

update

Thanks, John! I am very excited about this project, and also about the hardware, so I guess it shows :) 

As far as thermal monitoring goes, micsmc seems to be the utility to use there, and that link shows the location of 2 sensors, as well as a command line option, "micsmc --temp", that brings up a list of 7 temperature sensors: CPU, Memory, Fan-In, Fan-Out, Core Rail, Uncore Rail, Memory Rail. The micsmc link also mentions there is such a thing as "thermally throttled" Phi, but doesn't explain what what throttling does.

So, after another brief delay, I have the system and it boots CentOS 7! The relevant parts of the server I decided on, all from Newegg, are:

• CoolerMaster HAF X ("High Air Flow") case. I bought it specifically because it advertised support for 4 double wide cards. It's a monster, with 9 card slots -- it could probably handle 5 Phi's if it had a motherboard that supports it. So, no dremel needed here. The airflow is probably not enough for the Phi's, so I plan to open it up and blast it with my industrial fan (set to minimum speed; that thing is loud and powerful).
• An EVGA "Supernova" 1600W 92%+ efficient PSU. That should be good enough for 4 Phi's, although having a supernova in my apartment is a little scary. It's good that it's an efficient supernova, otherwise it would need more fans.
• ASUS Z10PE-D8 WS.
  • 64GB (4 x 16GB) of ECC RAM
  • To pair with the 64GB RAM, I decided to get a better CPU, an 8-core E5-2630v3 instead of the initial plan of an E5-2603v3.

This allows for an expansion with an additional E5-2630v3 and additional 64GB.

In case anyone runs into the same problem I did initially with booting, here is my experience. The Z10PE-D8 WS, as well as a number of other recent ASUS MB's, have a 2-digit 7-segment display on the motherboard which displays "Q-codes", diagnostic codes which turned out to be helpful when the system didn't boot! Mine got stuck at "62" which, some internet searching revealed, was actually "b2" -- "Legacy Option ROM Initialization". The video card was attempting to reserve memory from the motherboard. I didn't need a video card anyway, so I removed it, connected an old VGA cable to the onboard graphics, and it worked! It's also likely this could be changed in the BIOS.

So now it's time for me to dig into the software!

[Robert_F_2]

The case I've been looking at is the Thermaltake X9.  It's a big case and would allow me to install the cards without having to figure out how to rig something to support the weight of the cards.  These are very heavy with the heatsinks.

The case can fit an unreasonable number of radiators  and and an eatx motherboard at a very reasonable price.  About the same as a 4U server case and it stacks.

I will be ordering watercooling blocks from alphacool this week.

I'm currently building my box with twine and duct tape so I'm using 2 kits of 16GB ecc unbuffered, a couple of es E5 2665's on a salvage lenovo C30 thinkstation board with 2 phis.

I would like to trade up to the Z9PE-D8 WS board and more ram later with 4 PHI's and some more ram.

Note that on the Z10PE you need to price in DDR4 and V3 E5's.   On the Z9PE you can use DDR3 and V2 E5's.   As always it depends on your budget but untill DDR4 comes down significantly in price it is a lot cheaper to build on a Z9PE D8 Ws.

1600 watts sound right 300 per card and 300 more for you motherboard/cpus with a little margin for safety.

I'm really looking forward to my little kaggle / dev lab but it's taking many pay cheques to get up and running.

[ccnyc]

I also considered the Thermaltake X9. It looks awesome. The only reason I decided against it was due to my unorthodox cooling situation. Fabricating ducts to bring air from an outside fan into the case and across the cards would not be as easy as for the HAF X which may not need much to guide the air. If it were not for that, it looks like an awesome case.

You're absolutely right about the DDR4/v3 versus DDR3/v2 tradeoff being one of the key items to consider. Depending on what size RAM you're looking for the price difference could be substantial, or not. In my case I know my app performance will depend heavily on RAM performance, the granularity of the simulation will be bounded by the available RAM, and I will likely run some simulations fully on the Xeon E5-2630v3 before working with the Phi's, so I spent awhile looking for RAM, and eventually found 4x16gb of ECC DDR4 on newegg for $730. 4 x 16gb of ECC DDR3 would have been $600, which at least isn't that far off in this case.

But if having that much RAM isn't required, that's not a necessary expense. If I get to the point where I have a second server with its own 4 Phi's it's likely to have DDR3, an entry level processor, and a much smaller amount or RAM. That way, the first server can run any large-scale simulations, but the second still has 4 Phi's that can make quick work of anything that will fit on the cards.

[Robert_F_2]

I'm just following up on myself. I was never able to confirm whether there Lenovo D30 board was compatible as it definitely didn't like my b stepping ES E5's. The P9x79 ws, P9X79-E WS, z9pe-d8, z10pe-d8 are known to work so they are on my current shopping list. Building one of these with your own money is a pretty big project even with the deep discounts on the phi itself. I am still looking forward to first boot. I suspect knights landing may ship before I do.

Robert F. wrote:

The case I've been looking at is the Thermaltake X9.  It's a big case and would allow me to install the cards without having to figure out how to rig something to support the weight of the cards.  These are very heavy with the heatsinks.

The case can fit an unreasonable number of radiators  and and an eatx motherboard at a very reasonable price.  About the same as a 4U server case and it stacks.

I will be ordering watercooling blocks from alphacool this week.

I'm currently building my box with twine and duct tape so I'm using 2 kits of 16GB ecc unbuffered, a couple of es E5 2665's on a salvage lenovo C30 thinkstation board with 2 phis.

I would like to trade up to the Z9PE-D8 WS board and more ram later with 4 PHI's and some more ram.

Note that on the Z10PE you need to price in DDR4 and V3 E5's.   On the Z9PE you can use DDR3 and V2 E5's.   As always it depends on your budget but untill DDR4 comes down significantly in price it is a lot cheaper to build on a Z9PE D8 Ws.

1600 watts sound right 300 per card and 300 more for you motherboard/cpus with a little margin for safety.

I'm really looking forward to my little kaggle / dev lab but it's taking many pay cheques to get up and running.