• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 460
  • Last Modified:

GTK/C linux app has bad performence on usb file copying, how can i improve it?

I am writing an application for Linux in c thats using gtk 2 for its interface. The application takes files i have selected and moves them to a usb thumb drive while keeping track of transfer progress (updating a progress bar). The video files i am transferring can range from 50mb to a couple of gigs so i would like to optimize the transfer. I started threading the application but the issue seems to be in the file transfer portion. Using fclose on my input and output file seems to take a bit too long. Does anyone have suggestions on how i could speed this code up on the file transfer portion?
cpu: amd phenom2 x6 (more than enough for file transfer i assume)
hdd 32mb cache/ 7200 rpm
os : ubuntu 11.04
usb thumb drive is capable of 30mb write speed, using cp or gui i get around 10mbps, using my app i get around 5mbps but i believe thats due just to closing the out file.

struct FileStr
{
    char infile[128]; //full inputfile+ fullpath
    char outfile[128];
    int tid; //thread id
  
} FileStr;

pthread_mutex_t ftmutex = PTHREAD_MUTEX_INITIALIZER; //mutex to lock my couter for while loop(progress bar updating)
int opnFiles = 0;
int totalopnFiles = 0;
 struct FileStr files[100]; //the structure to hold input/output to hand over to children threads

void *fileThreads(void *dirLists)  /////////////// the part that seems to have a speed issue or needs a new file transfer function?
{

/////////////////////////////// start of what i think is the questionable area
    // take struck from calling thread and place values into useable strings
    struct  FileStr *threadStr;
    threadStr = (struct FileStr *) dirLists;
    int bb = threadStr->tid;
    char in[128];
    strncpy(in,threadStr->infile,128);// = (char*)threadStr->infile;
    char out[128];
    strncpy(out,threadStr->outfile,128);              /// input and out put path are now set
    //// basic fread/fwrite function 
    char buffer[BUFSIZ] = {'\0'}; /// don't really know  much about buffer(how to increase or decrease)
    size_t len = 0;
    FILE *in = fopen(in,"rb"); // open in and out
    FILE *out = fopen(out,"wb");
   // g_print("%s - after file \n",a);
    if(in == NULL || out == NULL) // check to make sure both locations are ok
    {
        perror("error occured opening");
      //  g_print("%s - error \n",a);
        in = out = 0;
    }  
    while((len = fread(buffer,BUFSIZ,1,in))>0) // while buffer has data to read write to outfile
   {
        fwrite(buffer,BUFSIZ,1,out);
    }
    pthread_mutex_lock(&ftmutex); // lock threads for decrement (to do with bar progress) // stops another file transfer thread from starting till this point(multiple writes 
     //seems to really bog down the transfer speed
    opnFiles--;
    totalopnFiles--;
    pthread_mutex_unlock(&ftmutex);
   
    fclose(in);
  
    fclose(out);// seems to be the slow part !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    g_print("thread exit");
  
 
   pthread_exit(NULL);
/////////////////////////////////////////////// end of what i think is the question area
}


void  *transferFiles(void *unused)  ///// first thread seperated from gtk/gdk thread
{
    g_print("\n lock Passed, starting transferMain \n");
    int slInt = 0; ///////////// used to calculate transfer size
    int anInt = 0; 
    int tgInt = 0; 
    slInt = spaceLeftSize();
    anInt = neededSize();
    tgInt = slInt - anInt;//////////////
    
    int i = 0;
    double percentage = 0.1;
    double percentageLast =0;
    int doubleSleep = 0;
    pthread_t ftThread[100]; // create threads so up to 100 files could be transfered if needed
    int ftT; //int for thread
    GtkTreeIter tv2iter; //iterator  //  GtkTreePath *path;
    gboolean valid; //(wether there is another step)
    valid = gtk_tree_model_get_iter_first(GTK_TREE_MODEL(store_tv2), &tv2iter); // step through a tree view for selected files// create struct array
    while (valid)
    {
        gchar *name; //variables for holding results of get
        int size; //  struct FileStr files;
        gtk_tree_model_get(GTK_TREE_MODEL(store_tv2), &tv2iter, COL_NAME, &name, COL_SIZE, &size, -1); //get item with size+name
        char instring[128] = "/lefs/";             //<-----------
        strcat(instring, name);
        char outstring[128] = "/media/";        /////            uses global variables to select the usb drive and stuff it into file path
        strcat(outstring, usbDir);
        strcat(outstring, "/");
        strcat(outstring, name);                        //<-----------
        
        strncpy(files[i].infile, instring,128);
        strncpy(files[i].outfile,outstring,128);
        files[i].tid = i;
	g_print("\n made handover -#%d \n",i);
        g_print ("\n instring = %s \n outstring = %s \n",instring,outstring); 
	i++;
	totalopnFiles++;

       valid = gtk_tree_model_iter_next(GTK_TREE_MODEL(store_tv2), &tv2iter); //gets next iter for next while loop run 
    }
    
        gdk_threads_enter();//update bar
	gtk_progress_bar_set_text(GTK_PROGRESS_BAR(bar), "Starting Transfer");
	gdk_flush();
	gdk_threads_leave();
	int handThreads = 0;
	for( handThreads = 0; handThreads < i; handThreads++)  // loop for actual spawning off file transfer threads(child threads)
	{
	     	pthread_mutex_lock(&ftmutex);
	    	opnFiles++;
	    	pthread_mutex_unlock(&ftmutex);
	    	
		ftT = pthread_create(&ftThread[handThreads],NULL,&fileThreads,(void *)&files[handThreads]);
		if(ftT)
		{
			g_print("ERROR; return code from pthread_create() is %d\n", ftT); exit(-1);
		}
		else
		{
		g_print("thread entered -#%d \n",handThreads);
		}
		    pthread_mutex_lock(&ftmutex);
		if(totalopnFiles > 0)
		{ 
		pthread_mutex_unlock(&ftmutex);
		 	while(opnFiles >= 1)
			{ 
			 	int current = spaceLeftSize();
			      	if (current > tgInt)
			      	{
					percentage = current - tgInt;
					percentage = anInt - percentage;
					percentage = percentage / anInt;
					if(percentageLast == percentage)   //sleeping function if transfer has not made much progress.
					{
						if(doublesleep > 2)
						{
						sleep(15);
						doublesleep = 0;
						}
						else
						{
						sleep(5);
						doublesleep++;
						}
					}
					percentageLast = percentage;
				 	g_print("%d > %d -wloop- perc - %f \n",current,tgInt,percentage);
				       // g_print("whiled    %d != %d \n",spaceLeftSize(),tgInt);
					gdk_threads_enter();
					gtk_progress_bar_set_fraction(GTK_PROGRESS_BAR(bar), (double) percentage);
					gdk_flush();
					gdk_threads_leave();
					sleep(5);
				}
			}
		}
		else
		{
    		pthread_mutex_unlock(&ftmutex);
		}
	  }
	  
	    g_print("\n Entering Percentage - out of threads to start \n");
	    percentageLast = 0;
	        pthread_mutex_lock(&ftmutex);
	    if(totalopnFiles > 0)
	    { 
		    pthread_mutex_unlock(&ftmutex);
		    while (opnFiles > 0)
		    {
			  sleep(3);
		      int current = spaceLeftSize();
		      if (current > tgInt)
		      {
			percentage = current - tgInt;
			percentage = anInt - percentage;
			percentage = percentage / anInt;
			if(percentageLast == percentage)
					{
					sleep(5);
					}
					percentageLast = percentage;
		 	g_print("%d > %d -wloop- perc - %f \n",current,tgInt,percentage);
		     
			gdk_threads_enter();
			gtk_progress_bar_set_fraction(GTK_PROGRESS_BAR(bar), (double) percentage);
			gdk_flush();
			gdk_threads_leave();
		      }
		      else
		      {
			  sleep(10);
			  gdk_threads_enter();
			  gtk_progress_bar_set_text(GTK_PROGRESS_BAR(bar), "Finishing - Do Not Remove Device");
			  gdk_flush();
			  gdk_threads_leave();
		      }
		    }
	    }
		 else{

		    pthread_mutex_unlock(&ftmutex);
		 }
	    sleep(1);
 	g_print("completed and waiting for disconnect");
        valid = gtk_tree_model_get_iter_first(GTK_TREE_MODEL(store_tv2), &tv2iter);
        while (valid)  // remove all the files that were transfered
        {
		gchar *name;  int size;//variables for holding results of get
		gtk_tree_model_get(GTK_TREE_MODEL(store_tv2), &tv2iter, COL_NAME, &name, COL_SIZE, &size, -1); //get item with size+name
		char delFile[256] = "/lefs/";
		strcat(delFile, name);
		unlink(delFile); //remove file
		gtk_list_store_remove(GTK_LIST_STORE(store_tv2), &tv2iter);
		valid = gtk_tree_model_get_iter_first(GTK_TREE_MODEL(store_tv2), &tv2iter); 
   	 }
   	 g_print("completed disconnect waiting for buttons");
   	 
    gdk_threads_enter();
    gtk_progress_bar_set_text(GTK_PROGRESS_BAR(bar),"");
    buttonLock(TRUE); //function to free other buttons on app
    gtk_widget_hide(cancelBtn);
    videoLock(FALSE); //turn  gstreamer back on
    comboChange(); //adjust usb and files available
    gdk_flush();
    gdk_threads_leave();
    g_print("FileTransfer thread Completed");
    pthread_exit(NULL);
    // g_thread_exit(NULL);
    //return(NULL);
}

void transferStart()//From the Commit BTN - starts the file transfer thread
{
    videoLock(TRUE); // stop gstreamer(set state to pause)
    buttonLock(FALSE);
   // gdk_threads_leave();
    gtk_widget_show(cancelBtn);
    gdk_flush();
    int ftT;
    pthread_t ftThread;
    ftT = pthread_create(&ftThread,NULL,&transferFiles,NULL);
}

Open in new window


My entire source code is a bit long to post but i think thats all that would be needed to figure why the process is coming out so slow. This is also my first app in Linux  and C so I'm sure that most of this is not best practice. If you need any more information please don't hesitate to ask. Thanks in advance experts for helping.
0
2sq
Asked:
2sq
1 Solution
 
Hugh McCurdyCommented:
I don't think fclose() is the actual slow part.  It's when you issue the fclose() that program's buffers are flushed.  If you put a     fflush ( out );    On the line before the close and time the fflush, if I'm right, it will appear to be the new bottleneck.

Please test that to make sure I'm right before we proceed.
0
 
Hugh McCurdyCommented:
Are you concerned about the actual write time to the usb drive (to the point it is unsafe to unmount) or just the time the program runs.

(You may or may not have noticed, depending on how the usb drive is mounted, that there's a delay after copying before it is safe to remove the device.)
0
 
satsumoSoftware DeveloperCommented:
This page might be helpful. http://davmac.org/davpage/linux/async-io.html.

As hmccurdy seems to be thinking, the delay on closing the file is probably the OS flushing buffers and waiting for IO to complete.  Asynchronous IO might allow your program to continue without blocking until completion.  I say might because I've never done AIO on Linux, only on Windows.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
Hugh McCurdyCommented:
There are potentially two buffers in play depending on how the drive is mounted.  The std io library does buffering of its own.  The OS might be doing other buffering.  Can't tell about the OS unless I know how the drive is mounted (O_SYNCW or something like that.  Going from very fuzzy memory.)
0
 
ssnkumarCommented:
First of all, you need to find out which function is taking more time.
So, put a time function before and after each function call which you think is consuming time.
The difference between these times give you the actual time spent in these functions.
So, that will tell you where exactly more time is taken.

After getting this info only, we can work on the problem and see if something can be done to improve the performance.

So, please modify your code, add time function before and after function calls that might be consuming time and dump the result here.
0
 
Hugh McCurdyCommented:
I'm pretty confident that, as written, fclose() appears to be taking the most time.  The standard io functions use buffered reads and writes.  The purpose is to avoid inefficient use of system calls (actually reading or writing a single character to a disk file, "right now").  Often this approach decreases overall execution time.  However, it has the effect of causing fclose() to look like closing the file takes a lot of time.  But what fclose() is really doing is flushing the buffers (fflush()) and then closing the file.  

If the time functions are added now, I predict fclose() will take a long time to finish (assuming a lot of data was written).  Once that time is known, I predict that if you put fflush() right before fclose() and time both of them that the bulk of the execution time will shift to fflush().
0
 
satsumoSoftware DeveloperCommented:
@hmccurdy, purely speculation, but I wouldn't be surprised if the hardware was buffering too.  Between data arriving from the USB bus and being written to the flash memory.  I guess thats why you get warnings about removing the USB drive too fast.  The computer may have cleared its own buffers but the drive could still be writing to the storage.
0
 
Hugh McCurdyCommented:
@satsumo, pretty easy to test my "speculation."  I'm also confident Linux is also buffering unless synchronous writes were requested.  I would not be surprised if there's a h/w buffer as well.  There is a hardware buffer for most internal disk drives.

I can share the following observation.  When I write a large amount of data from my computer to my Droid, it appears to finish fairly quickly.  But when I umount the Droid, there's a significant delay.  My guess is Linux has a big buffer for the USB drive and umount forces the flush.  The copy is done with the Linux cp command.
0
 
2sqAuthor Commented:
sry for taking so long to get back, thankyou all experts we now have a screaming file transfer!! Also multiple writes to the usb causes all sorts of issues and lock for anyone looking in the future not using usb 3.0 pen drives
0

Featured Post

Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now