Thread: Fine-Grained and Coarse-Grained actions - Help me understand

I've heard these terms used in parallel programming. In that context, the granularity refers to the amount you're able to split the original job up into.

For example, say you want to add up two vectors each with 1 million numbers. The coarsest possible grain size is to have everything in 1 job.

// coursest grain size
for(int i = 0; i < 1000000; ++i)
{
    result[i] = A[i] + B[i];
}

The finest grain size would be to have each grain (or task) add up one element.

// finest grain size
int thread_id; // get a thread ID somehow
result[thread_id] = A[thread_id] + B[thread_id];

And it's possible to have various grain sizes between these (for example adding up 10 elements/task, 100 elements/task, etc. etc.).

CPU's are designed to have "heavy weight" threads, i.e. they are capable of doing a lot of things at the cost of performance (speed). In the above example code, it would be silly to start a thread and have it add up two numbers then save the result. The amount of time it takes to start the thread and mange it exceeds the time it takes to add up a single element. The overhead is simply too great. However, if you end up with a trillion elements per job, the overhead becomes extremely small. If there are any idle processing units, then your program isn't utilizing the full computational potential of your computer and will run slower. The trick is to balance the right granularity size so that the overhead isn't unbearable, but also ensure that you are able to utilize as much of your computers resources effectively (there is also another issue of load balancing, but that's a slightly different issue).

To contrast this, in the context of the GPU, it has lots and lots of very light-weight threads. These threads aren't able to do as much as a CPU thread, but their primary advantage is that it's very easy to create lots of them very quickly. It's not unheard of to have a GPGPU kernel create and run thousands or tens of thousands of threads. Additionally, each core on the GPU is slower than that of the CPU. This lends GPU programming to benefit the most from having lots of very fine grain tasks (there are other reasons for this, too but I won't go into them here). On the GPU, adding just one element up or a handful of elements per grain/task is quite common.