multi thread writing

hi there,
I need some help with the multi threading, and i am sweatering with it right now. Thanks for your help.

Say, i everytime i retrieve 1000 records and i would want to split it to 2-3 threads, each thread would write to a seperate file, finally to combine the files into one. Any detailed working/example codes to show how do i create a thread, pass this array to thread, and wrting to files?
pigycAsked:
Who is Participating?

[Webinar] Streamline your web hosting managementRegister Today

x
 
makerpConnect With a Mentor Commented:
have you omitted some global variables.... if so these will need protecting with mutexes. when you have data that is used by more than one thread it is often the case that you will need to restrict access to one thread at a time other wise the data will get corrupt. look up synchronisation...

now if you are going to use synchonisation to ensure only one thread is writing to the array at any one time then what on earth is the point of having two threads doing it. all they will do is end up taking it in turns to write to the array, in actual fact it will probably take longer for two threads than one because if you are only using one you not need to worry about synchronsation and therefore wont have the overhead of it (which can be quite considerable)..

Paul
0
 
posternbCommented:
What compiler/platform?  Any particular thread library in mind/available?
0
 
pigycAuthor Commented:
on hp11, compiler? what command would i be able to check? I heard around that pthread might be the one unless there are other thread libs.
0
Evaluating UTMs? Here's what you need to know!

Evaluating a UTM appliance and vendor can prove to be an overwhelming exercise.  How can you make sure that you're getting the security that your organization needs without breaking the bank? Check out our UTM Buyer's Guide for more information on what you should be looking for!

 
dnatarajCommented:
# include <pthread.h>
# include <unistd.h>
# include <stdio.h>

void * WriteToFile1(void *arg){
     while(1)
        {
               //Read Record Function()
               //Write Record Function()

        }
}

void * WriteToFile2(void *arg){
     while(1)
        {
          //Read Record Function()
               //Write Record Function()
        }
}

main(){
     pthread_t pt1,pt2;
     pthread_create(&pt1,NULL,WriteToFile1,NULL);    
     pthread_create(&pt2,NULL,WriteToFile2,NULL);    
     
     while(1)
         {

}
0
 
dnatarajCommented:
# include <pthread.h>
# include <unistd.h>
# include <stdio.h>

void * WriteToFile1(void *arg){
     while(1)
        {
               //Read Record Function()
               //Write Record Function()

        }
}

void * WriteToFile2(void *arg){
     while(1)
        {
          //Read Record Function()
               //Write Record Function()
        }
}

main(){
     pthread_t pt1,pt2;
     pthread_create(&pt1,NULL,WriteToFile1,NULL);    
     pthread_create(&pt2,NULL,WriteToFile2,NULL);    
     
     while(1)
         {
                usleep(100);
         }

}


compile with -lpthread option
0
 
makerpCommented:
are you sure using more than 1 thread willl yield any real performance benifit. people often think that if they use more than one thread the task will be completed quicker; this is often not the case. the only real problems that can be solved with multithreading is getting over the delays with blocking calls; for example socket work etc. even this can be solved with async IO instead of blocking IO....

if you want to learn multi-threading then write an application that makes good use of it...

0
 
pigycAuthor Commented:
thanks for emailing my questions.
after reading 'makerp' email, i choose a path which is  different than my original ways:
launching two threads, each thread would retrieve a block of data from database and do some calculation and put the result into an array; finally each thread wrting the array result to a seperate file. Now everything has its merried way with one single thread and i saw each single file has correct records when running the program with single thread, until i run two threads together, then the results in two seperate files are wired, and it is hard to trace/tell what's happening inside.
i have the following pseudo code, anyone could light me that what is the  problem with this? why i don't have anything written out for these two files while single thread would write out correctly? Does one thread disturb/blocking/interfere the other thread's writing? BTW: if you have trick ways to trace, that would be very helpful. thanks

void *get_history_all( void *argv )
{
  /*get data and put them in a array*/
write_history_records(result, fp,  num_rows);
destroy_array(&result);
}

void write_history_records(char *result, FILE *fp, int num){
for(int i=0; i<num, i++)
fprintf(fp, "%s", ...);
}


int main(){
char temp1_file[256], temp2_file[256];
FILE *t1_fp, *t2_fp;
strcpy(temp1_file, "thread1.txt");
  strcpy(temp2_file, "thread2.txt");

  if(!(t1_fp = fopen(temp1_file, "w"))){
    printf("temp1_file open failed");
  }
  if(!(t2_fp = fopen(temp2_file, "w"))){
    printf("temp2_file open failed");
  }

 argval[0].fp = t1_fp;
 argval[0].fp = t1_fp;

ret_code = pthread_create(&first_thread, NULL, (void *(*)(void *))get_history_all, (void *)&(argval[0]) )
;
if (ret_code != 0)
  {
    printf("thread_create failed, error number: %d\n", ret_code);
    exit(1);
  }

ret_code = pthread_create(&second_thread, NULL, (void *(*)(void *))get_history_all, (void *)&(argval[1]))
;
if (ret_code != 0)
  {
    printf("arb_thread_create failed, error number: %d\n", ret_code);
    exit(1);
  }
/* wait for the first thread to terminate */
  ret_code = pthread_join(first_thread, NULL);
  if (ret_code != 0)
  {
    printf("first thread_create failed, error number: %d\n", ret_code);
  }
  fprintf(t1_fp,"%s","f1 finished");
  fclose(t1_fp);
  /* wait for the second thread to terminate */
  ret_code = pthread_join(second_thread, NULL);
  if (ret_code != 0)
  {
    printf("second thread_create failed, error number: %d\n", ret_code);
  }
  fclose(t2_fp);
  exit(0);
}

 
0
 
pigycAuthor Commented:
thanks for emailing my questions.
after reading 'makerp' email, i choose a path which is  different than my original ways:
launching two threads, each thread would retrieve a block of data from database and do some calculation and put the result into an array; finally each thread wrting the array result to a seperate file. Now everything has its merried way with one single thread and i saw each single file has correct records when running the program with single thread, until i run two threads together, then the results in two seperate files are wired, and it is hard to trace/tell what's happening inside.
i have the following pseudo code, anyone could light me that what is the  problem with this? why i don't have anything written out for these two files while single thread would write out correctly? Does one thread disturb/blocking/interfere the other thread's writing? BTW: if you have trick ways to trace, that would be very helpful. thanks

void *get_history_all( void *argv )
{
  /*get data and put them in a array*/
write_history_records(result, fp,  num_rows);
destroy_array(&result);
}

void write_history_records(char *result, FILE *fp, int num){
for(int i=0; i<num, i++)
fprintf(fp, "%s", ...);
}


int main(){
char temp1_file[256], temp2_file[256];
FILE *t1_fp, *t2_fp;
strcpy(temp1_file, "thread1.txt");
  strcpy(temp2_file, "thread2.txt");

  if(!(t1_fp = fopen(temp1_file, "w"))){
    printf("temp1_file open failed");
  }
  if(!(t2_fp = fopen(temp2_file, "w"))){
    printf("temp2_file open failed");
  }

 argval[0].fp = t1_fp;
 argval[0].fp = t1_fp;

ret_code = pthread_create(&first_thread, NULL, (void *(*)(void *))get_history_all, (void *)&(argval[0]) )
;
if (ret_code != 0)
  {
    printf("thread_create failed, error number: %d\n", ret_code);
    exit(1);
  }

ret_code = pthread_create(&second_thread, NULL, (void *(*)(void *))get_history_all, (void *)&(argval[1]))
;
if (ret_code != 0)
  {
    printf("arb_thread_create failed, error number: %d\n", ret_code);
    exit(1);
  }
/* wait for the first thread to terminate */
  ret_code = pthread_join(first_thread, NULL);
  if (ret_code != 0)
  {
    printf("first thread_create failed, error number: %d\n", ret_code);
  }
  fprintf(t1_fp,"%s","f1 finished");
  fclose(t1_fp);
  /* wait for the second thread to terminate */
  ret_code = pthread_join(second_thread, NULL);
  if (ret_code != 0)
  {
    printf("second thread_create failed, error number: %d\n", ret_code);
  }
  fclose(t2_fp);
  exit(0);
}

 
0
 
makerpCommented:
pthread_mutex_init(&mutex, NULL);     //initialize the mutex object
0
 
pigycAuthor Commented:
paul,
You have the points. yes if only thread could write to array, then there is no point to use two threads. I have something unclear--i don't think i have gloables unless the resizable array dynamically created in the get_history_all func would considered 'global' to two threads---I would imagine each thread would have their own access to get_history_all func and wouldn't share the array they each created in this function; now after they each created the array and fill in with data, it comes to write_history_record and passed in the array thread created to write to a different file. Would the array passed into the write_history_recored share by two theads? I wasnot quite sure on this--in single instance/thread, this wouldn't consider as global.

void *get_history_all( void *argv )
{
   /*first create resizeable array locally called */

 /*get data and put them in a array*/
write_history_records(result, fp,  num_rows);
destroy_array(&result);
}

void write_history_records(char *result, FILE *fp, int num){
for(int i=0; i<num, i++)
fprintf(fp, "%s", ...);
}


I define a struc to pass into the createthread function.
typedef struct loaderarg
{
  db_connection          *conn;
  int                           account_no_small;
  int                           account_no_big;
  FILE  *fp;
} LoaderArg;
0
 
pigycAuthor Commented:
i accidently hit the enter key.


in writing_history_record(   ){
/*first locally create resizeable array called*/
/*   RA    result;*/
/*filled in the array with data from db to 'result*/
/*pass this result to write_history_record   */
}

would this result array considered global to two threads?

I found anther thing which might add the reason why i don't get correct records to two output files: seems my db connection to fetech data from db is not thread safe--i need to figure out a way to do that. Ultimately,it is a learning process for me to touch multi threading. thanks
0
 
makerpCommented:
if the array is local to the function then each thread has its own array, if this is the case then forget my previous comment about synchronisation...

as far as your db_connection goes, i would wrap this with a mutex
0
 
pigycAuthor Commented:
Paul,
Thanks for your email. Yes i tested using the array not created dynamically using and it wrokds. I tested
db connection with mutex it works, and aslo i tested to open my db connection inside each thread(inside get_history_all function, and it works too). Things are narrowing down to what are globals(locals) to threading. As long as i don't mess this up, it works. And the box running my program has multi processors which is a good news. thanks for your help
0
 
makerpCommented:
glad to help....
0
All Courses

From novice to tech pro — start learning today.