Parallel programming without C++

[Tudor]

Hey there,

I'm working towards learning parallel programming, but I can't shake the feeling that I've missed the train on learning C++ well. We haven't done too much C++ in college and I must admit I haven't been too interested in it until now. Sure, I know how to write simple threaded C++ programs, but I've looked at some of the code in the recent topics about producer-consumer queue implementations and frankly, I don't understand much and I have the feeling that I'll never be able to write such code. Time is also an issue, as I am in my final college year and need to study for the school subjects as well.
So my question to you is: can I still write effective parallel programs without being a C++ pro? Is a language like C# enough? Or should I just drop it and start learning C++ seriously?

Thanks,
Tudor

[anthony_williams]

Firstly, the code that's been discussed for producer-consumer queues is low-level, highly advanced stuff. In general, you should not be writing code that is this low-level, so the fact that you don't understand all the ins and outs of the provided implementations is irrelevant. These things should be provided by high level libraries such as TBB.

You do not have to be a C++ pro to write effective parallel programs. You can write parallel programs in C#, or Fortran, or Erlang or Haskell or whatever other language takes your fancy. Even if you choose C++, you do not have to know all the low level details. What you do have to know is how to use the libraries you have available, such as TBB.

One thing that is the same regardless which language you choose is that you also have to get your "parallel programming" hat on. This means you have to think about what you are trying to accomplish, and how that overall task can be divided into potentially-parallel chunks. You have to think about the communication between these chunks --- what data they need access to, and where it comes from. If you can truly grasp these things then the details of how to implement this with a particular language and library are considerably simpler.

[Tudor]

Thank you for your answer.
To clarify, I have done quite a few parallelization exercises in C# and grown quite confortable with it. Also, I think that for starters I have a pretty good basic understanding of theoretical parallel programming issues.
My question is, can I basically avoid advanced C++ forever in my quest for good parallel programming?

[TimP]

Among the primary parallel programming models supported in Intel software tools, only TBB requires C++. Importance of C++ and TBB depends entirely on your application. For data parallel applications, OpenMP has regained importance, and C (with some C99) or Fortran may be more appropriate in such cases. Even some of the examples in TBB training are better suited to OpenMP.
I did find it important to gain a limited acquaintance with C++ after 30 years of Fortran and C, but I still use C++ less than the others.

[jimdempseyatthecove]

Tudor,

If you know C# you probably know enough to feel comfortable with simple programming techniques in C or C++.

Using C/C++ and simple programming techniques you likely finding it not too difficult to use OpenMP within your simple programs.

Multi-threading does have a set of quirks, but once you frame these quirks in your mind they will be relatively easy to deal with.

One of the first things you must learn is multiple threads read/modify/writing to the same locations need care.

var++;

for example (increment variable var) which reduces to one instruction on most processors would seem to be multi-thread safe, when in fact it is not. Two processors updating the same variable might end up performing

read(1)/read(2)/modify(1)/modify(2)/write(1)/write(2)

and in which case var gets ++ once when it should have been ++ twice.

This is not only true for C++ but for virtually every language for multi-threaded programming.

Learning to write class member functions for TBB is quite difficult, OpenMP is much easier.

As an intermediate consider examining the document file on http://www.quickthreadprogramming.com

This is a task based threading toolkit that can be use with little impact on a single threaded program. In addition to QuickThread there is Cilk and a few others. (wikipedia.com may list a few of the C++ addons as well as other language support for parallel programming QuickThread is in the process of being released now so it is not yet known.)

Jim Dempsey

[anthony_williams]

Quoting - Tudor

My question is, can I basically avoid advanced C++ forever in my quest for good parallel programming?

If you use another language then you will have no need for advanced C++.

If you use C++, then how advanced your C++ needs to be depends on your parallel programming tools --- there is nothing about parallel programming that inherently requires advanced C++ knowledge. For example, as Jim already mentioned, OpenMP does not use any advanced C++ features.

Of course, if you program in C++ then knowledge of advanced techniques may well help you write better (simpler and faster) programs, but this is the case even for single-threaded programming.

[Tudor]

So am I to understand that C++, considering openMP, tbb, etc. is not above other languages when it comes to parallel programming? To me, it seems that C++ has the most capabilities when it comes to parallel programming: more libraries, better low-level control, better control over compiler optimizations, etc.
I don't know, but it seems to me that if pitted against C++, C# falls short on many levels, even with its new TPL. Sure, you can use C++ functionality within C#, but that still counts as using C++, right?

[TimP]

As parallel programming would usually be undertaken as part of a comprehensive effort to improve performance, there is a strong emphasis on using languages which support native code compilation. C++ is not alone among such languages, and not alone in supporting interoperability with C++.

[jimdempseyatthecove]

In the parallel programming realm there is no particular dimensionality to constitute a vertical domain (above/beneath). Rather, it is more multi-dimensional.

Your initial queiry seemed to indicate you wanted simplicity over learning the full extent of learning/mastering C++. For that purpose you can stick with C programming style together with OpenMP. You still have to be careful of concurrent update access to shared variables and shared resources. But shared resources, and commonlyacceptedrules for handling these circumstances are an everyday experience. Take walking through a doorwayfor example. If someone reaches/enters the doorway before you, common rules generally mean you wait. Concurrent programming is no different. What is (slightly) different is learning what constitutes a potential adverse interaction point (e.g. doorway) and what are the possible solutions to avoid the adverse interaction. Often there is no single best solution for all circumstances.

For simplicity, consider the task of summing an array using simplified C

float sum = 0.0;
for(i=0; i sum = sum + Array;

AddingOpenMP in your first attempt would expose one of these adverse interaction points

float sum = 0.0;
#pragma omp parallel for
for(i=0; i sum = sum + Array;

The adverse interaction point is the doorway of "sum = sum + Array;"
Which has the potential for the read(1)/read(2)/modify(1)/modify(2)/write(1)/write(2) problem (and various other permutations of problems).

Also, you might think multiple threads using the same i with different values at the same time. OpenMP automatically separates the loop control variable in for statements. However, OpenMP does not automatically separates adverse interacting statements within the statement of {statements} following the loop control. For this there are other OpenMP directives.

float sum = 0.0;
#pragma omp parallel for reduction( + : sum )
for(i=0; i sum = sum + Array;

The reduction clause informs the compiler to generate separate variables for each thread all of which are named sum for private use within the parallel region. Then upon exiting the parallel regions combine the individual results (in private sum variables) into the single variable sum as it exists outside the parallel region (combined using + in this case).

While there is some learning required, for simplified uses the learning is not all that hard.

Start with C syntax. Learn to write a single threaded (simple)program. Then adapt it gradually for multi-threaded programming using OpenMP. As you add parallel regions to your working program, expect the program to break. Figure out (or ask here) what is (or might be) the adverse interaction point. Fix the problem, and move on. In a relatively short time, you will attain a proficiency sufficient enough to write reasonably good (simplified) parallel programs.

Jim Dempsey

[Tom_Spyrou]

Quoting - anthony_williams

Firstly, the code that's been discussed for producer-consumer queues is low-level, highly advanced stuff. In general, you should not be writing code that is this low-level, so the fact that you don't understand all the ins and outs of the provided implementations is irrelevant. These things should be provided by high level libraries such as TBB.

You do not have to be a C++ pro to write effective parallel programs. You can write parallel programs in C#, or Fortran, or Erlang or Haskell or whatever other language takes your fancy. Even if you choose C++, you do not have to know all the low level details. What you do have to know is how to use the libraries you have available, such as TBB.

One thing that is the same regardless which language you choose is that you also have to get your "parallel programming" hat on. This means you have to think about what you are trying to accomplish, and how that overall task can be divided into potentially-parallel chunks. You have to think about the communication between these chunks --- what data they need access to, and where it comes from. If you can truly grasp these things then the details of how to implement this with a particular language and library are considerably simpler.

[Michael_K_Intel2]

Quoting - Tudor

So am I to understand that C++, considering openMP, tbb, etc. is not above other languages when it comes to parallel programming? To me, it seems that C++ has the most capabilities when it comes to parallel programming: more libraries, better low-level control, better control over compiler optimizations, etc.
I don't know, but it seems to me that if pitted against C++, C# falls short on many levels, even with its new TPL. Sure, you can use C++ functionality within C#, but that still counts as using C++, right?

Hi!

If you're not interested in squeezing out the last bit of performance of your machine, it would be worth to try out Java, too. For Java, you can try out most parallel programming models as well. As you know you can program plain old threads in Java, being the most low-level programming approach. You have a higher level, going to Java's task API (java.util.concurrent). I know of MPI-bindings for Java (I can look up the links if you want).I have written an OpenMP compiler that understands Java code enriched with OpenMP parallelization hints.With JCilk you get the Cilk programming model (sort fork/join model for tasks) and so on and so on.

Let me know if you're interested and if you want more information on this.

Cheers,
-michael

[Tudor]

Quoting - Michael Klemm, Intel

Hi!

If you're not interested in squeezing out the last bit of performance of your machine, it would be worth to try out Java, too. For Java, you can try out most parallel programming models as well. As you know you can program plain old threads in Java, being the most low-level programming approach. You have a higher level, going to Java's task API (java.util.concurrent). I know of MPI-bindings for Java (I can look up the links if you want).I have written an OpenMP compiler that understands Java code enriched with OpenMP parallelization hints.With JCilk you get the Cilk programming model (sort fork/join model for tasks) and so on and so on.

Let me know if you're interested and if you want more information on this.

Cheers,
-michael

Thanks for the reply. I'm still in the experimentation phase of my parallel programming career hehe, so anything new is welcome. I've only done some basic threading in Java with threads and threadpools. That openMP compiler sounds good, as I've decided to pick up openMP in C as well.

[DweeberlyLoom]

Quoting - Tudor

Thanks for the reply. I'm still in the experimentation phase of my parallel programming career hehe, so anything new is welcome. I've only done some basic threading in Java with threads and threadpools. That openMP compiler sounds good, as I've decided to pick up openMP in C as well.

You might also consider the .NET framework if you are running windows. MS has a "parallel extension" that is available as a seperate download for the current framework and will be standard for the v4.0 framework. For more info see http://msdn.microsoft.com/en-us/concurrency/default.aspx

Mono (if you are not windows)appears to also have an implementation http://tirania.org/blog/archive/2008/Jul-26-1.html

[jose-jesus-ambriz-me]

About the first question I'm agree with dweeberlyloom, .net framework ( C# ) it's a good options for beginners and with the new framework at 2010 will be better.

[yuryserdyuk]

You can try also MC# programming language - www.mcsharp.net

[robert-reed]

Quoting - Tudor

Thanks for the reply. I'm still in the experimentation phase of my parallel programming career hehe, so anything new is welcome. I've only done some basic threading in Java with threads and threadpools. That openMP compiler sounds good, as I've decided to pick up openMP in C as well.

If you're just learning, you might want to follow along ona new blog sequence I've started. Perhaps I can make the prospect of using C++ a little less scary as I take apart, step by step, the process of designing, writing and tuning a parallel program, trying as I make my way to step in as many holes as I can find to outline them and show how to get around (or out of) them. C++ library code certainly can be very dense and obtuse as is needed to squeeze through all of the hoops imposed by multiple architectures and compilers and thread safety concerns. Most of that is "hidden" (you shouldn't need to look at it), hopefully presenting a simpler model at the user level. So far I've been setting up the problem and I'm about to step into the deep (multi-threaded) end. Come check it out!

[vu64]

Quoting - Robert Reed (Intel)

If you're just learning, you might want to follow along ona new blog sequence I've started. Perhaps I can make the prospect of using C++ a little less scary as I take apart, step by step, the process of designing, writing and tuning a parallel program, trying as I make my way to step in as many holes as I can find to outline them and show how to get around (or out of) them. C++ library code certainly can be very dense and obtuse as is needed to squeeze through all of the hoops imposed by multiple architectures and compilers and thread safety concerns. Most of that is "hidden" (you shouldn't need to look at it), hopefully presenting a simpler model at the user level. So far I've been setting up the problem and I'm about to step into the deep (multi-threaded) end. Come check it out!

I think it is a good place to learn about C++ concurrent programming. Also pick up a good C++ introduction like Programming Principles and Practices using C++ by Bjarne Stroustrup or Accelerated C++ by Andrew Koenig.

[laurenebsary]

Quoting - anthony_williams

Firstly, the code that's been discussed for producer-consumer queues is low-level, highly advanced stuff. In general, you should not be writing code that is this low-level, so the fact that you don't understand all the ins and outs of the provided implementations is irrelevant. These things should be provided by high level libraries such as TBB.

You do not have to be a C++ pro to write effective parallel programs. You can write parallel programs in C#, or Fortran, or Erlang or Haskell or whatever other language takes your fancy. Even if you choose C++, you do not have to know all the low level detail
s. What you do have to know is how to use the libraries you have available, such as TBB.

One thing that is the same regardless which language you choose is that you also have to get your "parallel programming" hat on. This means you have to think about what you are trying to accomplish, and how that overall task can be divided into potentially-parallel chunks. You have to think about the communication between these chunks --- what data they need access to, and where it comes from. If you can truly grasp these things then the details of how to implement this with a particular language and library are considerably simpler.

[mahmoudgalal1985]

Quoting - Tudor

Hey there,

I'm working towards learning parallel programming, but I can't shake the feeling that I've missed the train on learning C++ well. We haven't done too much C++ in college and I must admit I haven't been too interested in it until now. Sure, I know how to write simple threaded C++ programs, but I've looked at some of the code in the recent topics about producer-consumer queue implementations and frankly, I don't understand much and I have the feeling that I'll never be able to write such code. Time is also an issue, as I am in my final college year and need to study for the school subjects as well.
So my question to you is: can I still write effective parallel programs without being a C++ pro? Is a language like C# enough? Or should I just drop it and start learning C++ seriously?

Thanks,
Tudor

You can use java and OpenMp