Parallelism types from Implicit to Explicit

1. Instruction Level parallelism
2. Compiler assisted parallelism
3. Programmer guided, Compiler assisted parallelism
4. Programmer guided, automatic multi-threading
5. Multi-threaded programs (manually developed)
6. Multi-process applications (manually developed)

Instruction Level parallelism
This is what is provided at the processor level, when you compile the serial code, parallelism is automatically extracted by the microprocessor. The CPU tries to runs concurrent instructions in parallel. This is a pure hardware solution, future articles will discuss how instruction level parallelism is achieved and the challenges developers must concur in order to fully utilize this parallelism.

Compiler assisted parallelism
Still in the implicit parallelism category, the compiler and the CPU work together to achieve parallelism. The compiler identifies concurrent instructions while compiling and tags them with OP codes. When the CPU executes the instructions the ones with OP tags are run in parallel. The difference here is the hardware overhead is decreased because the software is taking on some of the load of extracting parallelism. This requires the compiler to be highly optimized for the CPU, for example ICC, the Intel C Compiler.

Programmer guided, Compiler assisted parallelism
Here is where we start to fall a little over to the explicit parallelism, but mostly it’s implicit. This is just like compiler assisted parallelism, but the programmer helps the compiler by reorganizing parts of the code. I will be writing an article soon that deals with some cool code reorganization, which will improve your code big time! Usually, this process is done via a process called profile driven optimization. Profilers are tools that can show you bottlenecks in your code.

Programmer guided, automatic multi-threading
This one falls more into the explicit parallelism, but has some implicit as well. The idea here is to use a programming language or a compiler that is extended/enhanced with special constructs that can be used by the programmer to tag different areas that concurrent. A good example of this is OpenMP. The reason it’s automatic mulit-threading is because the compiler will handle the work of creating, synchronizing, and destroying threads for you. However, scope of gain is limited to shared-memory architecture.

Multi-threaded programs (manually developed)
Now we are getting into the full explicit parallelism category. These programs are basically mulit-threaded. Most languages you use will have a way to create and use threads. If you have a mulit-core machine, it’s a given to use multi-threads which will utilize the different cores. The programmer is responsible for the extra overhead of synchronizing critical sections and avoiding deadlocks and race conditions. For most programmers, it’s often challenging to develop good multi-threaded programs.

Multi-process applications (manually developed)
This is the last type and the most difficult to get right. This is fully explicit parallelism and the program typically runs on multiple processes. Each process works on a different computer (computes are interconnected). Note, each process may contain mulitple threads. Again, it’s up to the programmer to keep track of all the overhead of allocating resources and handling synchronization. This is the current favored solution on most supercomputing clusters.

That concludes this article. Remember the idea of this article was to get you a little info into the different types. Future articles will go even further into some of these types and discuss how we can optimize our code to get some big improvement. Questions or comments are always welcomed!

1. A different computer thats networked or interconnected
2. Another microprocessor on the same computer
3. Another core on a microprocessor
4. A seprate ALU or part of ALU on a processor

This idea of parallelism is extended and we have different types of parallelism. Imagine a spectrum if you will. In this spectrum, one extreme end will be implicit parallelism and the other extreme is explicit parallelism.

Implicit parallelism in its basic form is parallelism that is automatically or semi-automatically realized. Meaning the microprocessor and the compiler work together to extract parallelism. Note, that the scope of what you gain are limited to a single processor or single machine. Explicit parallelism is the other extreme, so parallelism here is realized manually. The programmer is responsible for developing the program to run in parallel. Typically, for anything worthwhile, would require considerable programming efforts. However, the gain be scoped to multiple machines or supercomputing clusters!

You might be asking, why is explicit parallelism so difficult to code? If you are not asking yourself this question, pretend you just did! It’s tough because most of the time the serial running program that you created is hard to break up into multiple small chunks of code. Basically, parallelism strives to effectively utilize concurrency in code to reduce the overall computational time required to complete processing.

By concurrency, I am talking about the lack of dependency or relationship between two tasks. Identifying concurrency is what makes explicit parallelism so hard. If you understood everything to this point, then you have the basics of parallelism and the challenges. Part2 of this article will discuss the different types of parallelism, the stuff between our imaginary spectrum we described earlier.

I have big interest in HPDC and this section of my blog is all about HPDC. Most of my examples are with C unless stated otherwise. I hope you learn from this section and don’t hesitate to ask questions!

1. char
2. short
3. int
4. long

Article outline
I’ll talk about each of the following operations and give some examples on how to use each.

1. Bitwise NOT (~)
2. Bitwise AND (&)
3. Bitwise OR (|)
4. Bitwise XOR (^)
5. Shift Left (<<)
6. Shift Right(>>)

Bitwise NOT (~)
This is an unary operator. It simply filps all ’0′s to ’1′s and ’1′s to ’0′s. Here is a simple example:

unsigned byte x = 10;
x = 00001010
~x = 11110101

Bitwise AND (&)
This is a binary operator. Just follow the truth table. Both bits have to be true to get a resulting true bit. Anything else is false. Here is a simple example:

unsigned byte x = (10&8);
10  = 00001010
8   = 00001000
x   = 00001000

Bitwise OR (|)
This is a binary operator. Just follow the truth table. Here, just one of the bits has to be true to get a resulting true. Here is a simple example:

unsigned byte x = (10|8);
10  = 00001010
8   = 00001000
x   = 00001010

Bitwise XOR (^)
This is a binary operator. Just follow the truth table. Basically it flags all bits that are different. Here is a simple example:

unsigned byte x = (10^15);
10  = 00001010
15  = 00001111
x   = 00000101

Shift Left (<<)
Keep in mind that when you do one of the shift operators the size of the data type does not change.
A left shift, moves the bits to the left inserting ’0′s in the right most bit for every shift. The left most bit is discarded. This is effectively multiplying the number by 2. Note if you use too many shifts you could end up with a negative number. This is possible if the data type is interpreted as 2′s complement. Here is a simple example:

unsigned byte x = 4 << 2;
4   = 00000100
x   = 00010000
x = 16

Shift Right (>>)
A rightshift, moves the bits to the right inserting ’0′s in the left most bit for every shift. The right most bit is discarded. This is effectively dividing the number by 2. Note if you use too many shifts you could end up with a zero. Here is a simple example:

unsigned byte x = 4 >> 2;
4   = 00000100
x   = 00000001
x = 1

That wraps up the bitwise operators. If there is anything I forgot, let me know and I’ll make the updates. I hope you learned something in this read. Remember these are operation which the processor can execute very fast. For example, if you every need to multiple or divide by 2, it best to use the shift operations. Not only will you code execute faster, but also save you processor some energy!