• C

Discussion: Coding and debugging Multithreaded code.


I'm putting together a tool right now that is using a multithreaded architecture. Its actually in C++ but I'm asking it here because I dont think that is relevant.

As this is probably the first time I actually have an input on the overall design and architecture of a multithreaded app I thought I would canvas some experts! :-)

I am looking more for architectural design and general methods rather than specific techniques for specific problems.

The areas I want to cover are:

1. Design.

What do you avoid or encourage in the overall design? If one thread reads from a structure and one writes to it, how do you ensure that they can both work together in harmony without losing the benefits of multithreading, especially with complex structures such as lists and trees?

2. Debugging.

There are loads of both obvious and wierd race conditions that can sneak up and grab you. Many can take hours of work to track down. From the obvious log file sharing issues to the complex 'walking a tree while it's being updated'. How do YOU go about tracking down the actual problem, avoiding the old 'well it must be somewhere in here so I'll just close the door on the whole structure with a mutex'.

It's a bit wierd to hit a breakpoint and see the other threads carry on running! How does one freeze the whole system at a break point so you can investigate the state and history of all threads at the exact moment of the problem?

What logging methods do you use to ensure that traceback is possible without losing the bug by changing the thread timings?

3. Testing.

As most threading issues arise through rare occurrences, what techniques are good for exposing the issues early in the development cycle?

LVL 16
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

> If one thread reads from a structure and one writes to it, how do you ensure that
> they can both work together in harmony without losing the benefits of multithreading

Independend from the kind of synchronisation, I think you definitely need a single function to access shared data (or two, one for reading, one for writing). It is easier to debug, and if there is something wrong with the performance, there is only a single spot to improve.
Just write clear code, with one exit point per function, where you can bottleneck all the locks and unlocks.
Maybe run a thread info window, that displays some interesting info about each thread.
Kent OlsenData Warehouse Architect / DBACommented:
Hi Paul,

When I write multitasking applications, I solve a LOT of headaches by writing them one at a time, and writing them so that nearly all of the data accesses are to local variables.  This might sound silly, but it's so easy to get caught up in the "I need a variable to keep track of this count so I'll just put it in the globals block" mentality.  The next thing you know, you're updating "Count" from several threads, with disastrous results.

And I've never been tasked with writing a multi-threaded application in C.  (I know several people that have, but not me.)  I'll move on to C++ for the project and let the controls and containment of the C++ wrappers do the heavy lifting.  My own rule is that every thread object is defined as a class.  This way the variables ARE local.  If I want to write to other threads I have to explicitly reference a public variable or one that considers this task a "friend".

Something that might help in C coding is to write every thread as a separate source file.  This way, liberal use of the STATIC modifier let's you create thread variables much like the C++ class.

Debugging is a huge issue, too.  I use the Borland C++ IDE which makes stepping through the code easy, but you do have the issue of the other threads.  If they can run while I'm debugging a thread, I let them.  Otherwise, I build in a mechanism to sequence the threads.  When the thread that I'm debugging completes a pass, it increments a counter and returns control.  The next task picks up the count change, runs a pass, and repeats the process.

Most issues are due to poor design, not poor implementation.  Sit and write the specifications and controls before you start coding and you're a long way ahead.  Writing the code without designing it (or have a LOT of experience at this) is a sure way to shoot yourself in the foot.

Become an IT Security Management Expert

In today’s fast-paced, digitally transformed world of business, the need to protect network data and ensure cloud privacy has never been greater. With a B.S. in Network Operations and Security, you can get the credentials it takes to become an IT security management expert.

PaulCaswellAuthor Commented:

Thanks for that. Lots of good stuff.

May I summarise?

1. If it CAN be local, make it local.

This is good practice in any language and environment and having this habit ingrained in a multithreaded environment is a must. Totally agree.

2. Use the highest level language you can.

I dont object to this, in its place. However, in my opinion, one of the powers of multithreading is the ability to tune for speed and efficiency through the interconnectedness of the threads. If you take the language too high-level you may lose some of these benefits. Obviousy, if the process itself is innately parallel then you are right but when threading is being used primarily for a speed advantage then this can be an architectural mistake.

The tool I am writing is a parser at its center. I will be using multiple threads to run multiple state machines over the data list, some creating new lists as they go for others to follow. I am using STL for the data structures and currently raw C for the threads. I may move the threads into objects in time but it is, as always, the access to the shared objects that causes the most problems.

3. Make it a deliberate act to access shared data.

This is exactly the kind of stuff I was looking for. An architectural guideline.

4. Most issues are due to poor design, not poor implementation.

Agreed! Can you think of some examples of bad design? Or good design?

Great feedback! :-)

Only 10,000 more points !!! Not long now !!! :-)

Kent OlsenData Warehouse Architect / DBACommented:

>> Can you think of some examples of bad design?

The 1971 Pinto comes to mind.
PaulCaswellAuthor Commented:
PaulCaswell, you are guru in C, I know and it looks like questionarie for others :)
But I have few words also.
My recommendaions are based on Unix kernel design and on real kernel code.
There are good links where you may read more about multithreaded programming.
I prefer Sun Microsystems docs: http://docs.sun.com/app/docs/doc/806-6867?q=Multithreaded

Also there is a 'Maurice J. Bach' book 'Unix design' where concurrency issues and workarounds are suggested (for example how to avoid deadlocks).

For list structures there is a common recommendation to avoid data corruption (of course it doesn't avoid race condition):
- use one lock for list head (entire list)
- use one lock for each element
this technique is used in disk/network cache in kernel where cache is a bundle of hashed lists

If your thread requires more then one lock:
- use the same order of locks for lock aquisition in all threads (in all control flows) to avoid deadlocks
- use the order opposit of acuisition to release locks

Kent OlsenData Warehouse Architect / DBACommented:

Just a thought, but Oracle defines two levels of locking.  User data (rows, pages, tables, etc) use a lock at the appropriate level.

Processes interlock internal structures by a process called latching.  It's a short term (few micro/millisecond lock).

For the purposes of this discussion, we could do the same.  Intertask communication is achieved with a latch, user data is controlled with a lock.

I advise you to treat multithreading as heavy, scary, dangerous voodoo magic.  (Your question indicates you may already see it that way.)  As you are aware, threading bugs can be very difficult to find.

Therefore, minimize the distribution of complexity among threads.  It's better to have one thread that has most of the complex logic and let the other threads do simple things than to have several threads doing complex things concurrently.

Similarly, make the interfaces between threads as minimal and simple as you can.

Serialize processing with event or message queues.  If you have an object where multiple threads may be calling different functions at unpredictable times, don't try to have those functions synchronize their operations on the object if you can avoid it.  Instead, if you can, have those functions queue jobs to be processed by the object one at a time serially.

As Kent suggested, encapsulate synchronization.  Instead of having an unwritten design rule, "thou shalt lock this object before accessing it," design a function with locking built in, that is the only possible way to access it.

Reviews or inspections are good for catching bugs early in the development cycle.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.