receiving

huh? I need to send trunks of each file and multiple files in a directory and subdirectory. I don't send whole file at a time.

Hi Begin,

What do you mean "trunks of each file"?  


Lines 133 - 206 can all be replaced with that short loop.  You're doing an awful lot of processing that you don't need to do.

If you want to add verification and sanity checks you can, and it certainly makes sense to do that.  But having a chunk of code to send a 2K block, another to send a shorter last block, and another to send a file shorter than 2K is over-complication.



Kent

For example for it need to transfer a 4 GB file. It will send each 2048 byte portion at a time. That's why from line 133-206 chop the file portion.

Hi Begin,

I'm not sure what you're saying there.  Can you tell me a little more?

Once you've identified a file to transfer, most of the logic is done.  Actually sending the data can be as simple as reading from the file an writing the data to the socket.


Kent

here in my idea to transfer files in a directory and its subdirectory from client to server:
  + Read all file name  in directory and find total of files to be sent from client to server
  + send total file to server. this will help to determine how many iteration in for loop in receiving at server side
  + for each file, send filename and its size to server. file size help to determine how many iteration  to send portions in in for loop in order to send complete file. Similar to server side.  
         
 

Hi Begin,

Ok.  Let's look at your 3 bullets in big-picture terms.

- The first bullet is what you're trying to accomplish.  That is, you want to copy an entire folder and all subfolders to another system.
- The second bullet tells the server how many items the client is going to send.
- The third bullet is how you intend to send each file.

The first bullet has nothing to do with the code.  That's your goal.

The second bullet and thirds bullets should be restructured.  The program logic would look something like this:

main () {
  send_all_files ();
}

send_all_files () {
  list_all_files ();
  send_file_count ();
  for (idx = 0; idx < filecount; idx++)
    send_file (file[idx]);
)

send_file () {
}


Your code doesn't need to be any more than about 1/2 of of what you have now.


Kent

 

 

The client side code will be easier to code and understand if you separate the application logic (what you're trying to do) from the system logic (how you're going to do it).

To make that happen, the sendFile function can be broken up into smaller pieces.

Let me restruct my code in following format now. tks.

    main () {
  send_all_files ();
}

send_all_files () {
  list_all_files ();
  send_file_count ();
  for (idx = 0; idx < filecount; idx++)
    send_file (file[idx]);
)

send_file () {
}

PS: just wonder whether you see what cause mismatch. hihi.

Hi Begin,

I looked briefly, but quickly saw that things were already too complicated for the relatively simple task that you're trying to do and didn't look any further.


Kent

HI Kdo,
 I see. I am restruct my function now. I will post back when I finish it. Tks. I am so desperated now :)

Don't be desperate.  :)

We'll make this happen!

I just finished to reorganise the code.  So please advise  what error i have made. tks a lot.
client.c
server.c

Hi Begin,

Going back to the send() function, it still reduces to the code below.


Kent

void sendfile( SOCKET sock, string filename)
{
        FILE *fp;
        fp= fopen(filename.c_str(),"r");// t is the filename string
        long BytesSent;
       
	if(fp == NULL)
	{ 
		printf("error to open %s",filename.c_str());
                exit(1);
  } 
	// determine file size 
        fseek(fp, 0, SEEK_END);
        int filesize = ftell(fp);
        rewind(fp);
      
        string t = filenamefilesize(filename, filesize);      
              
        char *pch = (char*)malloc(sizeof(char) *t.length() +1);

        memset(pch,'\0',sizeof(char)*t.length()+1);

        strcpy(pch, t.c_str());
        printf("File name and its size just sent!!!. No content here\n");

        send(sock, pch,t.size(),0);
        printf("just send file name and its size !!!!!\n");
              

    buffer= (char*)malloc(sizeof(char) *BUFFER_LEN);
  while ((result= fread (buffer, 1, BUFFER_SEND, fp) > 0)
  {                
    BytesSent + send (sock, buffer, result, 0);
  }
  free(buffer);             
  fclose(fp);
}
}

Select allOpen in new window

I am appreciated your motivation. You simplify sendfile() to very succinct level. I don't know why I never thought about it.  

First feedback is it's amazing. i am speechless. It send all files now. I gonna test with file in subdirectory.
Please explain what i did wrong in my fumble code :)

Looking at your OP code (sorry, EE went off-line), your list includes a folder. If you have n files in a single folder, then you should be able to transfer the n files. But as you try to go up to the parent, that appears to be where the problem occurs.

Shouldn't your list only include files and not folders?

I am running ddd debugger now. i like it. but there is the problem. for example when i try to see the buffer value in server side. I always see '\000' < repeat 2048 times>, but i can see the value result is 2048.
 It can't work with the subdirectory.

 >>Shouldn't your list only include files and not folders?
could you please elaborate more? tks.

In your OP, you had this LOC:
>>   //  cout<<"<LIST> has :"<<t<<endl;
Uncommented, it produces the entire list of files to transfer. But, on that list was the sub-folder. Is that intentional? I didn't think you wanted folders to be on this list since you don't want to transfer a folder.

<LIST> has :/home/ubuntu/program/client
<LIST> has :/home/ubuntu/program/server.c
<LIST> has :/home/ubuntu/program/client.c
<LIST> has :/home/ubuntu/program/. 4096
<LIST> has :/home/ubuntu/program/test
<LIST> has :/home/ubuntu/program/huh/text_edit_commands.html
<LIST> has :/home/ubuntu/program/huh/text_compare_session_settings.html
<LIST> has :/home/ubuntu/program/huh
<LIST> has :/home/ubuntu/program/server
<LIST> has :/home/ubuntu/program/test.c

do u mean i should not include "." in the list?

I want to transfer all files and folders too :) For example FolderA has 8 files, and a FOLDERB. inside FOLDERB, there are 5 files. So i want to transfer all :) l

I see that huh is a folder, and it is on the list. When you do with this entry?

oh i want to transfer "huh" folder and all files inside that folder too. It's similar like we copy and paste folder .

Ok. If you have one subfolder, then keep in mind that the problem seems to appear only after completing the transfer of all the files in the subfolder.

Hi Begin,

I was going to suggest that you drop all of the data that your server code receives and just print out what you think that it should be doing.


Kent

I gonna remove all display.im offline for 8hrs.tks

The client sends the name of the subfolder to the server. When the server receives this subfolder name, the size is 0. The server opens this subfolder as a file. Shouldn't the server be made aware that this name is a folder and then, if necessary, create the folder?

Server should creae folder like copy and paste hihi

Ok, never did that way before to create a folder. I see that a file called "subfolder 0" was created. Hmm, it should have "drwx" attributes, but it only has "rw" attributes.

I had a chance to look at latest code patching in kdo's last code post. Stepping through the debugger in client and server, both exited normally with all files created. Then executed both without debugger and got a seg fault. So, some memory clobbering is going on somewhere.

I guess list have some problems

How are you testing. I am on one system and change ubuntu to localhost. So the files get created in the correct folder since we provide the full pathname, and the folder already exists. Are you testing with two systems?

For testing in same pc,we test easily bc we appended filesizein filename.list loax all file before writing so it can aviid conflict.tks

Right, by adding the filesize, we get a new filename - good idea.

I was wondering what you saw regarding the folder issue. I did not get a folder, but rather a file having the same subfolder name with a "space 0" appended to the name. Did you see this behavior.

fopen does not create a directory.

But you can do it several ways. Take a look at:
     http://rdsrc.us/EM87E2

But, now you have the issue of having the server distinguish between a file and a folder. And then there is the order in the list. If the server receives the full pathname where some of the folders do not exist, then you will have to parse the pathname into the set of folders, and verify that each exists, and once you find one that does not exist, then you can create it plus all of its subfolders (from the rest of the pathname.

If taking that approach, then having the actual folder name on the list is redundant. On the other hand, if you require that all the folders of a pathname already exist, then the folder name should be received before the files and folders that are under it. Then your server can test for the folder existence, and if needed, it can create the folder.

I see what you meant. I thought linux treat evertthimg like file.thar t I assume fopen could  create folder.i must do research aboyt it.im of computer for 6 hrs.tks for tellung me what issues need to study.

Hi Begin,

Something to keep in mind is that there is little correlation to the size of the buffer or object that you send() and the size of the buffer or object that you receive().

That is, the client can issue a send of 10 bytes and another of 20 bytes.  Nothing in the network layers say that the data will be received in buffers of 10 and 20 lengths.  If you call receive () with a buffer size of 30 or more bytes, you may well get both objects that you sent in a single read.

One way around that is to send something in the front of each block with details about the block.

The example below shows 1 (of many) possible ways to do this.  The key to this is that the server is flexible.  The sending of an unknown number of blocks isn't rigid and inflexible.


Kent

#define MAXBLOCK 2000
  enum {NOTHING, FILENAME, DATA};

  int   BlockType;
  int   BlockLength;
  char   Buffer[MAXBLOCK ];

//  Send the file name

  BlockType = FILENAME;
  send (sock, &BlockType, sizeof (int), 0);
  BlockLength = strlen (Buffer);
  send (sock, &BlockType, sizeof (int), 0);
  send (sock, Buffer, BlockLength, 0);

//  Send the data

  buffer= (char*)malloc(sizeof(char) * MAXBLOCK );
  while ((BlockLength = fread (Buffer, 1, MAXBLOCK, fp) > 0)
  {                
    BlockType = DATA;
    send (sock, &BlockType, sizeof (int), 0);
    send (sock, &BlockLength, sizeof (int), 0);
    send (sock, &BlockType, sizeof (int), 0);    
    BytesSent + send (sock, buffer, result, 0);
  }
  free(buffer);             
  fclose(fp);

Select allOpen in new window

I will study your guide more careful bc I cant access to pc now.but I see the problem more clear now.file path issue when create with different folder n write in it.

I have 1 idea to test it.let put all test data in 1 folder   and store data after servee receive in another folder.by that way, no need to worry about testing issue hihi

back to action :) i am reading  all feedback carefully. tks a lot.

It's very useful (I would say important) to remove as many warnings as possible, so that when a serious warning appears while maintaining your program, then you will actually quickly notice it rather than have to scan a dozen warnings.

To get rid of many warnings, you can replace the printf statements with either perror() or your error() which in turn calls perror() and exits.

Here is another example (same with t1):
>> if( t2= close(sockfd) == -1)
>>    error(...)
Just add the parenthesis as you see fit. Note that == has precedence over = , and changing the order will cause t2 to have different values when there is an error.

Re: the subfolder issue when going remote..
Before trying to write code, it might be beneficial for you to jot down pseudo-code for review.

You do not have to include folders in the list as they are inferred from the full pathnames that you are providing. But now you have to parse and test and if necessary, create a directory.

Alternatively, you could send first a tree-list message that represents all the required folders.

After reading all comments carefully, I want to verify those bullets:
   + Remove all warnings and use error().
   + In server, receiving block with current implementing is cluttered up. I should use file size as decremental flag until it equal 0 . ( use similar approach from client side)  
   + Use system call to make folder
   + First, send a tree-list message for all require folders and create those folder if not exist. Then I can use the current code.
   + In the code Kdo post, I have question
 

//  Send the file name

  BlockType = FILENAME;

  send (sock, &BlockType, sizeof (int), 0);
 
  BlockLength = strlen (Buffer);

  send (sock, &BlockType, sizeof (int), 0);  << why do we need to send BlockType second time?

  send (sock, Buffer, BlockLength, 0);

//  Send the data

  buffer= (char*)malloc(sizeof(char) * MAXBLOCK );

  while ((BlockLength = fread (Buffer, 1, MAXBLOCK, fp) > 0)
  {                
    BlockType = DATA;
    send (sock, &BlockType, sizeof (int), 0);
    send (sock, &BlockLength, sizeof (int), 0);
    send (sock, &BlockType, sizeof (int), 0);  <<<< why do we need to send BlockType second time?

    BytesSent + send (sock, buffer, result, 0);
  }

Tks a lot.

     

The idea of messaging is for the sender and receiver to know exactly how many bytes are being sent so that the receiver reads exactly what is needed for the message and not accidentally reading part of the subsequent message. One approach is to have a consistent fixed length message header structure which includes the number of bytes being sent in the body of the message. Another way used for short messages of known width is to just send the data in binary form (this is basically a brief message header). Numbers that fit into a 32-bit integer is an example of this latter case, so you don't need the length.


Messages:  

1. Number of files to be transferred

You should either send a message: "messge length" + "number of files ASCII string" or you can just send a 16- or 32-bit integer that represents the total number of files.

2. Filename + Filesize

This is the message that you already send prior to sending the trunk of the file. You should preceed this message with the number of bytes in the message so that server reads exactly those number of bytes. (I sent a 32-bit int, and then was able to transfer all files including those in subfolders.)

3. Trunk of the File

Since you already sent the filesize, it is not necesary to send the filesize again.However, the above is used just to get your program working. The other advice given is to make your program have a consistent messaging pattern so that there are no special rules for each message. I would try to get a functional program working first, and make step-wise improvements in the software engineering aspect of your development with complete regression test for each improvement (make sure you backup your previous working version).

  For my level, I think i don't need to worry about Endianness and host/network ordering.
   Let me work two things:
         + fix string parse by using strrchr
          +on the the improvement of message. By sending number of files to transfer and filename + filesize. In server, receiving block with current implementing is cluttered up. I should use file size as decremental flag until it equal 0 . ( use similar approach from client side). Give me around 1 hr to finish those tasks.    
 
     After I improve them, I can work on directory. tks a lot.  

my little progress. It works with only files. Please take a look.
Should I move on to build the tree-directory list now? tks for your patient. I am so slow.
client.c
server.c

I'll try to take a look at your code as soon as I get a chance (today, if possible; certainly tomorrow).

>> Should I move on to build the tree-directory list now?
Not sure how familiar you are with tree structures. If you are and believe you have a good handle on how you would define a tree-directory message, then you could post pseudo-code. But there are a number of steps:

(1) You have to collect the data in some container that you define, and then
(2) convert that container to a single tree-directory message.
(3) on how to parse this message and
(4) actually ensure that the directory tree is created properly before you receive the set of file transfer messages.

>> I am so slow.
Not really. :)

I need to open a new question regarding to building tree-directory. I think i can handle the tree from with your guide in another question. I will be back at midnight .  
 Tks for your motivaion and help.

Hi BeginToLearn,

I got involved here primarily because of the subfolder issue, which you are starting to work in the new question you opened. (But I would ignore that problem until you fully understand the remaining issues here.)

I also made some suggestions that are related to the segment faults stemming from your file transfer. Now, kdo is providing you with an excellent methodology for sending messages. So, please continue with kdo in this regards. In the end you will have a working program in this question (with only subfolder issues remaining).

Could you check to see whether you uploaded the latest version of your client and server code. I tried compiling the latest version, but got warnings of which at least one represents a serious functional error (i.e., not just a compiler nuisance warning). I think you should strive to post only no-warning files. If you are unable to remove the warnings, then you can always ask a new question to remove compile/build warnings/errors, and you will get a quick response.

In your uploads, I did not see the change related to this important comment:

>> Filename + Filesize

Note that this comment is for the quick method regarding what kdo already spoke about. Namely, you only want to read exactly the number of bytes that you know you should read and none more. kdo's approach is general in dealing with this issue. As long as you have time, go with kdo's approach.

A heads-up for you. There are issues in the new server approach w.r.t. the number of bytes read and written. It would be beneficial to your learning experience if you could take the time to try to figure out what is wrong yourself.

If you run up against a wall, then by all means let kdo know the problem, and he'll give you a good hint.

I got your program to work in my quick and dirty method. But, to prepare for commercial programs, I'd go with a more uniform messaging approach.

what does  "w.r.t" present?
 My upload should have no error in compiling.  I forgot to change  code reflecting the important comment  
    >> Filename + Filesize      
     >>This is the message that you already send prior to sending the trunk of the file. You should precede this message with the number of bytes in the message so that server reads exactly those number of bytes. (I sent a 32-bit int, and then was able to transfer all files including those in subfolders.)

 Let me do it now.

w.r.t. = with respect to

Ok, but there are other issues w.r.t. the server not writing the correct number of bytes to the file.

Ok, don't forget to use the gcc or g++ -Wall compiler option

omg, now i recompiled and executed them again before working on that important comment, I go segmentation fault. haiz

You have to fix the other problem I talked about also. :)

I see now. It only works with files inside current directory only. If any other directory exist there, it has segmentation core.
 So i only need to work on " the important comment" hihi

Did you mean this:

   Client side:
        // at this place: add send filename + filesize
        while (result= fread (buffer, 1, BUFFER_SEND, fp) > 0)
        {                
           
                BytesSent + send (sock, buffer, result, 0);
        }

>> BytesSent + send (sock, buffer, result, 0);
makes no sense to me.

The serious problem is on the server side.

When you test, you should verify that the file created matches the file from which it is derived (length check is good; a diff is even more assuring - but I've found with your code that if the lengths are the same, the diff is also good.)

Server:
    while (result= fread (buffer, 1, BUFFER_SEND, fp) > 0)
        {      
            BlockLength = result;
                send( sock, &BlockLength, sizeof(int), 0); // sending info about size of sending packet
                        
                send (sock, buffer, result, 0);
        }

Client:

while( size >0)
                {
                        read( newsockfd,&BlockLength, sizeof(int));// get size of packet receiving
 
                  result= read(newsockfd,buffer,BUFFER_SEND);
                        if( result <BlockLength)
                        {
                       error("reading trunk is failed\n");
                  }
                        result= fwrite (buffer,1,BUFFER_SEND,fp);
                        if( result != BUFFER_SEND)
                        {      
                             error("writing error\n");
                        }
                        memset(buffer,'\0',BUFFER_LEN);
                        size= size - BUFFER_SEND;
                }        

Is that what you mean the server need to verify how many bytes it need to receive?

Heh, you labelled the Server: and Client: backwards.

I see what you are doing, and I think it is beginning to capture the spirit of what kdo was saying; namely, to include a message header that informs gives the length of the body of the message sent (not "packet" since that term is used in tcp/ip).

With this approach, you wouldn't even need to keep track of size since the client could then send a special EOF message.

But, this isn't the approach you or I were taking before. And this approach still suffers from the same problem that is in the server. You need to do some debugging. It isn't obvious (obviously, or you would have seen the problem); but once you figure out what the problem is, then you will have a good chance of fixing it.

So, first identify the basic problem - the segment fault is just a symptom.

Perhaps, start off with no sub-folders and just have two files in the current folder. That should not work.

ouch !!!. right now my current directory has 4 files: client, client.c, server,server.c. It works . can u pls tell me which part has potential/hidden bugs?

I'll be happy to test the scenario you described. But I'll need your version - one that has no warnings when using -Wall -ggdb options.

let me fix those warnings with -Wall  option. give me some minutes. tks.

here you are :)
client.c
server.c

Here is server error:

filesize is : 5605
size is 5605
reading trunk is failed
: Invalid or incomplete multibyte or wide character

Select allOpen in new window

No error messages from client.

As kdo said, there are timing issues to be considered. Try this. Set a breakpoint in the server at
     for( int i = 0; i <totalfile;i++)
Then run the client without any breaks.
Then step through the server looking at every variable change in value. See if the program still works for you.

Let me debug now

I can see some thing  weird:
   + BlockLength always  equa 0  afterread( newsockfd,&BlockLength, sizeof(int))
   + result is alway equal 1 after ead(newsockfd,buffer,BUFFER_SEND)

however, result equal 2048 after e (buffer,1,BUFFER_SEND,fp);
 I need your explanation here.
  Server crashes because of strncpy().

I can see some thing  weird:
   + BlockLength always  equa 0  after read( newsockfd,&BlockLength, sizeof(int))
   + result is alway equal 1 after read(newsockfd,buffer,BUFFER_SEND)

however, result equal 2048 after fwrite (buffer,1,BUFFER_SEND,fp);
 I need your explanation here.
  Server crashes because of strncpy().

So, you're saying that this code works well in some scenarios and not in others. There are multiple problems and then multiple questions.

We should wait for kdo since I think he first hit the nail on the head (although the issue of timing was implicitly discussed in your previous question).

Given the difficulties in that sometimes you have no problems and sometimes you do, then I guess you should not take any of my shortcuts, but instead start modifying your code using kdo's methodical approach for every message. Right now, you are handling some messages correctly; but others are not (and yet you say they give you the correct results sometimes).

One thing about subfolders.. After I got your code running, then it doesn't matter if there are subfolders. The files in them transferred OK in the single subfolder that I had. What went wrong was that the subfolder itself appeared as a file (with space 0 appended to the name). But since the folders existed in my test environment, then the transferred filename fopen'd correctly.

I would read kdo's text carefully in http:#35169917 and ask questions about it if there is something that you do not understand.

You started adopting the idea, but not completely. After you adopt the idea for every data transfer, let's see if the problem goes away.

Sorry . I just came back from supermarket. I read kdo comments and asked about :

  send (sock, &BlockType, sizeof (int), 0);  << why do we need to send BlockType second time?
  send (sock, &BlockType, sizeof (int), 0);  <<<< why do we need to send BlockType second time?


 in http:#35169917
I think we need to find another way because the code is unstable.Let's wait kdo comeback.

Too much problems :) It's so complicated than I thought.  

So, when is your deadline?
Would it help if you didn't send any BlockType at all. Usually we need to distinguish between one message and another when any message type can come in. In your case, the messaging between client and server is lock-stepped. It is true however that sending the BlockType does add an extra measure of sanity checking to verify that you really are in lock-step. Given the issues you are having (and the fact that it doesn't take much to get your program working), maybe for the first phase, you leave out the BlockType.

So, would you find it easier to proceed if you just dealt with BlockLength? If so, then use that for every message that you send.

deadline is 24. I am going to work and come back at 1 am. I will think about it. my cellphone can access internet so i can read whenever you post :). appreciated your  patient and help.

How do I use "ubuntu" without having to change your code to "localhost"?

Is that 24 midnight?
Do you ever sleep?

Don't forget, if you have a specific problem that you can describe simply, then if you ask a new question, it will be easy for others to help 24 x 7.

Ioften go to sleep at 1::300 or 2am n wake up at 7 am.im very exhausted but I saw u online early.i dont want to lose time.it is due 24  at midnight

I gonna open new question about this toniye bc it is too long now

I just run clienr n server on ubuntu as guess of virtur box on win 7.so I have no idea

Ok, try to make the question complete as possible, as self-contained as possible, AND as simple as possible.

I tried to install Oracle's VirtualBox on XP, but I got a warning saying that the software was uncertified and could mess up my system; so I figured I better learn more about the dangers more.

I also was going to install a CD Clone so that I could install an ISO without having to burn a CD, and got the same message.

I checked on kdo. It looks like he is inactive today on EE - maybe his day off.
You are actually very close to a fix. In the meantime, you can define your tree hierarchy message.

I let netbook instaling same ubuntu on virtualbox on windows xp.so farso good.its running at home for 6hrs already.
very cool bc we cab have same system.tks for ur time and understanding.

I hear a power nap is rejuvenating.
    http://en.wikipedia.org/wiki/Power_nap

I defintely need nap. I also need to do research on many stuffs.that is y u see I ask multiple questions hihi