WDB
asked on
Create socket in Process1 then pass it to already running Process2 and have Process2 use it
The Setup: I have a client/server environment where Process1 listens for connections on a certain port on a linux server. When a connection request is received from a client process, a new socket is created by accept() and then Process1 is forked. The forked process calls execlp to start Process2 and passes the new socket as a command line argument to Process2. Process2 then communicates via the new socket with the client process who made the original connection request. Process1 continues to listen and repeats this process for each connection request it receives from a client process.
The Issue: There are times where the Client Process and Process2 will temporarily lose their connection with each other (Both processes will continue to exist even when disconnected) . An example of this is when a mobile client machine goes outside the range of a wirless lan and then comes back in. I want to come up with a way that I can get the client process to reconnect to Process2 and pickup where they left off without having to shut down and restart the client process.
Ideal Solution: 1. Client process connects to Process1 and another socket is created via accept()
2. Client process tells Process1 that it has been disconnected from process 2 (Client Process will have the process id of Process2)
3. Process1 then somehow takes the new socket and sends it to Process2 and then process 2 starts using it as it's means of communicating with Client Process. (This one's the kicker and the point to this question)
So, how can I create a socket in process1 and get it to process 2 for use?
I have seen something similar to this done in windows environment with WSADuplicateSocket() and WM_COPYDATA but did not find anything in Linux that mimicks this.
The Issue: There are times where the Client Process and Process2 will temporarily lose their connection with each other (Both processes will continue to exist even when disconnected) . An example of this is when a mobile client machine goes outside the range of a wirless lan and then comes back in. I want to come up with a way that I can get the client process to reconnect to Process2 and pickup where they left off without having to shut down and restart the client process.
Ideal Solution: 1. Client process connects to Process1 and another socket is created via accept()
2. Client process tells Process1 that it has been disconnected from process 2 (Client Process will have the process id of Process2)
3. Process1 then somehow takes the new socket and sends it to Process2 and then process 2 starts using it as it's means of communicating with Client Process. (This one's the kicker and the point to this question)
So, how can I create a socket in process1 and get it to process 2 for use?
I have seen something similar to this done in windows environment with WSADuplicateSocket() and WM_COPYDATA but did not find anything in Linux that mimicks this.
Hi WDB,
I would suggest that you consider threads as an option ..
Your main process can spawn off a pre-configured number of server threads ... Each capable of servicing one client ...
When a client connects to the server, the main process accepts the connection and hands the client to one of the free threads ...
This brings in the issue of maintaining thread states ... Each thread can either be
free --- ready to get a new client
or busy --- working on a client
or waiting --- lost the connection unceremoniously so waiting for a reconnect
When a client is handed over to a thread, the main will pass back the server thread id to the client ... Client can use this id to uniquely identify the thread it was working with ...
Last problem is that of restoring the connection to the same state of data exchange ... For this you need to define your exchange in such a way such that it is stateful and can be remembered by client as well as server ... This is very much specific to the task you are accomplishing
The main advantage of this approach is that you save on all that IPC that you will have to do for communicating between parent and children ... this is more efficient as well as cleaner
cheers
sunnycoder
I would suggest that you consider threads as an option ..
Your main process can spawn off a pre-configured number of server threads ... Each capable of servicing one client ...
When a client connects to the server, the main process accepts the connection and hands the client to one of the free threads ...
This brings in the issue of maintaining thread states ... Each thread can either be
free --- ready to get a new client
or busy --- working on a client
or waiting --- lost the connection unceremoniously so waiting for a reconnect
When a client is handed over to a thread, the main will pass back the server thread id to the client ... Client can use this id to uniquely identify the thread it was working with ...
Last problem is that of restoring the connection to the same state of data exchange ... For this you need to define your exchange in such a way such that it is stateful and can be remembered by client as well as server ... This is very much specific to the task you are accomplishing
The main advantage of this approach is that you save on all that IPC that you will have to do for communicating between parent and children ... this is more efficient as well as cleaner
cheers
sunnycoder
It would also help to have a transaction ID sent out with every reply from the server. The client quotes the last transaction ID everytime it send in a request. The transaction ID should start from 1 everytime a new process is created. It has to keep incrementing for every transaction. So, based on the transaction ID, the "listening server" can identify at what stage the client left off.
ASKER
Thanks for the responses. I originally went down the path of the thread idea and I'm confident it would've worked but there was one problem....routers and firewalls. Our customers will be running several clients (sometimes hundreds). Using the thread approach would require our customers to open more ports on their routers/firewalls and that isn't going to be possible or acceptable. Also, when the disconnecting occurs I will only be concerned with the client sessions that were idle and not actively communicating with server sessions. If a client becomes disconnected and then is still trying to be used by someone then it will eventually exit and this is ok and acceptable at this point. I don't want to take on the task of trying to recover from that sceanrio at this time. I just would like to be able to resync up to idle client/server combos that have become disconnected.
I am still looking for an example of how I can pass a socket from one process to another that is already running.
I'm currently efforting the possibility of using my original listening process to receive "reconnect" messages and then sending a generic "reconnect" message to the specified server process such that when the server process receives this message it will begin listening on a new port for a reconnect from the client and then, if received, will continue as normal. Of course, this solution is much more complicated management wise then just passing a socket but I think it's workable. Fun Fun!!
I am still looking for an example of how I can pass a socket from one process to another that is already running.
I'm currently efforting the possibility of using my original listening process to receive "reconnect" messages and then sending a generic "reconnect" message to the specified server process such that when the server process receives this message it will begin listening on a new port for a reconnect from the client and then, if received, will continue as normal. Of course, this solution is much more complicated management wise then just passing a socket but I think it's workable. Fun Fun!!
Have you looked at Unix Network Programming Volumn 1, Second Edition, section 27.9, Richard Stevens, Prentice Hall. Examples from book, ftp://ftp.uu.net/published/books/
This section is titled: "TCP Preforked Server, Descriptor Passing".
Also, check out this link: http://www.linuxdocs.org/HOWTOs/Secure-Programs-HOWTO/sockets.html. There is more information about passing file descriptors using "Unix Domain sockets".
I hope this helps,
Anthony.
This section is titled: "TCP Preforked Server, Descriptor Passing".
Also, check out this link: http://www.linuxdocs.org/HOWTOs/Secure-Programs-HOWTO/sockets.html. There is more information about passing file descriptors using "Unix Domain sockets".
I hope this helps,
Anthony.
ASKER
I have developed a solution to this scenario that uses my original thread approach in such a way that only one additional port is required. If I could get the file descriptor passing to work correctly then no extra ports would be required and that would be ideal. Unfortunately I can't afford the time to program this just to find out that it won't work. However, if I can be assured that it will work then I will take the time to add it. I have researched passing file descriptors in linux but I cannot find any evidence that it will work in my situation.
My Current Solution: 1. Each server program starts a separate thread that listens on a specified port for a reconnection request
2. When a disconnect occurs, the client app connects to the original listening program at the already opened and available port
3. The listening program then connects to the reconnect thread of the server process that it needs to reconnect to. (Since they listener and the
server
program are running on the same machine I will not require them to be opened on firewalls which is very important).
4. When the server's thread is connected to it begins listening at a new fixed port (This is the port that I will have to add with respect to firewalls
etc..) . Every server process thread use this same port and therefore I will only be able to do 1 at a time which is ok. (The original listening
process will manage the order in which reconnects occur and make sure that only 1 occurs at a time)
5. The client app will be attempting to connect to this fixed port and when the connection occurs, the original socket in the server program will get
replaced with the newly connected socket and things will continue as normal.
I have tested this solution and it works and, at this point, is what I'm going to run with. It's also transparent to the main part of the server process that gets reconnected and this is another HUGE plus.
Optimal Solution: Building on my current solution, if I could use file descriptor passing to get the socket that the client is using when communicating to the listening process to the server process it needs to reconnect with then I wouldn't need to add that extra port. This brings me back to the orignal question and I have outlined the scenario again below.
Scenario:
Client Process C connects to Listening Process A. Listening process A accepts connection creating new Socket 1 and then forks. The child process that is spawned makes a call to execlp() and creates Process B and Socket 1 is passed to Process B as a command line argument. Child process exits(). Process C then communicates to Process B via Socket 1 until program(s) exit at sometime in the future.
Next, connection is lost for some reason between Process C and Process B. Process C reconnects to Listening Process A. Listen Process A accepts connection creating new Socket 2. HERE IS THE PROBLEM: Need To Get Socket2 From Listening Process A to Process B so B can replace it's disconnected socket with it and continue communications with Process C . I'm not even sure if this is going to be possible with passing of file descriptors but I'm hoping it is.
If I were to implement passing of file descriptors this is what I would technically do (Don't know if it would work but maybe someone can tell me if it would or give me an example of what would work? : )
For each connection request to the listening program I would create a pair of fd's via socketpair(). Then in the forked child process I would pass one of these fd's to execlp so the created server process would receive it in it's command line. The forked child process then exits. The server process that is created creates a thread and then sits on a recvmsg using the fd that was passed in. When the original listening process receives a reconnect request it accepts the connection creating a new socket and then calls sendmsg on the fd that goes with the server process that needs to be reconnected to. The new socket that has been created is what sendmsg will send. The server process thread recvmsg gets a message and extracts out the socket and replaces it's original socket handle and communication resumes. So, will that work? Will I be able to use the new socket created when listening app recieves a reconnection request from a client, pass it to the server app via sendmsg() and then use it in the server app to communicate with the client?
At this point if the passing of file descriptors doesn't work I'm not sure anything else will. I have a solution to run with but since my original question is how to get that socket passed I will be very happy to award all points to anyone providing an example or a link to an example of how I could get the above scenario to work or, a detailed explanation of why it won't work.
Thanks!
My Current Solution: 1. Each server program starts a separate thread that listens on a specified port for a reconnection request
2. When a disconnect occurs, the client app connects to the original listening program at the already opened and available port
3. The listening program then connects to the reconnect thread of the server process that it needs to reconnect to. (Since they listener and the
server
program are running on the same machine I will not require them to be opened on firewalls which is very important).
4. When the server's thread is connected to it begins listening at a new fixed port (This is the port that I will have to add with respect to firewalls
etc..) . Every server process thread use this same port and therefore I will only be able to do 1 at a time which is ok. (The original listening
process will manage the order in which reconnects occur and make sure that only 1 occurs at a time)
5. The client app will be attempting to connect to this fixed port and when the connection occurs, the original socket in the server program will get
replaced with the newly connected socket and things will continue as normal.
I have tested this solution and it works and, at this point, is what I'm going to run with. It's also transparent to the main part of the server process that gets reconnected and this is another HUGE plus.
Optimal Solution: Building on my current solution, if I could use file descriptor passing to get the socket that the client is using when communicating to the listening process to the server process it needs to reconnect with then I wouldn't need to add that extra port. This brings me back to the orignal question and I have outlined the scenario again below.
Scenario:
Client Process C connects to Listening Process A. Listening process A accepts connection creating new Socket 1 and then forks. The child process that is spawned makes a call to execlp() and creates Process B and Socket 1 is passed to Process B as a command line argument. Child process exits(). Process C then communicates to Process B via Socket 1 until program(s) exit at sometime in the future.
Next, connection is lost for some reason between Process C and Process B. Process C reconnects to Listening Process A. Listen Process A accepts connection creating new Socket 2. HERE IS THE PROBLEM: Need To Get Socket2 From Listening Process A to Process B so B can replace it's disconnected socket with it and continue communications with Process C . I'm not even sure if this is going to be possible with passing of file descriptors but I'm hoping it is.
If I were to implement passing of file descriptors this is what I would technically do (Don't know if it would work but maybe someone can tell me if it would or give me an example of what would work? : )
For each connection request to the listening program I would create a pair of fd's via socketpair(). Then in the forked child process I would pass one of these fd's to execlp so the created server process would receive it in it's command line. The forked child process then exits. The server process that is created creates a thread and then sits on a recvmsg using the fd that was passed in. When the original listening process receives a reconnect request it accepts the connection creating a new socket and then calls sendmsg on the fd that goes with the server process that needs to be reconnected to. The new socket that has been created is what sendmsg will send. The server process thread recvmsg gets a message and extracts out the socket and replaces it's original socket handle and communication resumes. So, will that work? Will I be able to use the new socket created when listening app recieves a reconnection request from a client, pass it to the server app via sendmsg() and then use it in the server app to communicate with the client?
At this point if the passing of file descriptors doesn't work I'm not sure anything else will. I have a solution to run with but since my original question is how to get that socket passed I will be very happy to award all points to anyone providing an example or a link to an example of how I could get the above scenario to work or, a detailed explanation of why it won't work.
Thanks!
I guess you can SO_REUSEADDR could do the trick for you ... just 4-5 lines of code which you should be able to try
read these man pages carefully
man setsockopt
man 7 socket
I shall post a detalied reply soon
read these man pages carefully
man setsockopt
man 7 socket
I shall post a detalied reply soon
grrrrrrr
>I guess you can SO_REUSEADDR could do the trick for you ... just 4-5 lines of code which you should be able to
I guess SO_REUSEADDR could do the trick for you ... just 4-5 lines of code which you should be able to
>I guess you can SO_REUSEADDR could do the trick for you ... just 4-5 lines of code which you should be able to
I guess SO_REUSEADDR could do the trick for you ... just 4-5 lines of code which you should be able to
ASKER
I'm not sure how SO_REUSEADDR would work for me. I'm actually using it in my current solution for the 1 port I'm using for reconnection requests to keep it responsive in case multiple clients are trying to reconnect but I'm not sure how it could help in getting my socket passed off the way I need it. I have tried the file descriptor passing method I mentioned above. I used socketpair(), forked the process and then passed one of the fd's to execlp hoping that the created process would be able to communicate over it but it didn't work. I'm still playing around around with it to see if there is anything I can do to get it to work. If I can't get it to work soon I'm just going to finalize the solution I've already implemented which is working well (other then needing an extra port). I will keep this question around for a bit and will be happy to award the points to anyone that can show me how to pass a socket to another process in linux.
Thanks!
Thanks!
Hi WDB,
First apologies for the late reply. I am still low on content to give you anything like a piece a code which I have tried, but here are a few ideas and estimates which may be of some help.
Next an idea for a different architecture .... One of your major contraints is that you cannot open several ports on the customer's firewall ... So I was thinking, why not have a single point of communication in your application. The main server thread forks out several threads (and not processes) ...
Create a unix domain socket for communication between each thread and the main thread. Each thread has a unique thread-id and each client that connects to the server is supplied a unique client-id. Here you may need to adjust the message format for communication between client and server to include the server thread id and client id in each message (going in any direction).
Each client communicated directly with the main server only ... The server just reads the server thread id field and does a lookup in a table to find the unix domain socket for that server thread and passes the message to it. Likewise to send a message to client the server thread passes the complete message to the unix domain socket ... main thread collects it from there and passes it to appropriate client.
Now the disconnection scenario. When a client requests a reconnect, it would know the server thread id to was talking ... If you are doing some timeo-outs and other house keeping stuff in the program, then you can devise a reconnect message... Client sends the reconnect message to the main server thread who as usual passes it to the appropriate server thread and your server thread knows that it has got reconnected.
The approach seems to fit your contraints and seems to be extensible. Only concern is the load on the main thread under very heavy load. The scalability may need to be tested here.
Second, I am sorry for my ignorance, but I fail to see any advantage that forking separate processes offers over spawing separate threads ... What you need to do with threads will have to be done with processes too ... It is just that you have additional IPC problems to deal with.
>Our customers will be running several clients (sometimes hundreds). Using the thread approach would require our
>customers to open more ports on their routers/firewalls and that isn't going to be possible or acceptable.
You can connect to one main thread and it can easily pass the socket to the server thread !!!
Third, I was hopeful of using SO_REUSEADDR in this scenario because this socket option would let another process bind to the socket and listen to the socket if no process is already listening to it. So you accept the new connection and pass it to server process who does all the send/recv operations on it ... I am not sure if this would work but it just may. That is the only way I can think of by which server process may be able to bind to a socket without creating it first.
Good luck
sunnycoder
First apologies for the late reply. I am still low on content to give you anything like a piece a code which I have tried, but here are a few ideas and estimates which may be of some help.
Next an idea for a different architecture .... One of your major contraints is that you cannot open several ports on the customer's firewall ... So I was thinking, why not have a single point of communication in your application. The main server thread forks out several threads (and not processes) ...
Create a unix domain socket for communication between each thread and the main thread. Each thread has a unique thread-id and each client that connects to the server is supplied a unique client-id. Here you may need to adjust the message format for communication between client and server to include the server thread id and client id in each message (going in any direction).
Each client communicated directly with the main server only ... The server just reads the server thread id field and does a lookup in a table to find the unix domain socket for that server thread and passes the message to it. Likewise to send a message to client the server thread passes the complete message to the unix domain socket ... main thread collects it from there and passes it to appropriate client.
Now the disconnection scenario. When a client requests a reconnect, it would know the server thread id to was talking ... If you are doing some timeo-outs and other house keeping stuff in the program, then you can devise a reconnect message... Client sends the reconnect message to the main server thread who as usual passes it to the appropriate server thread and your server thread knows that it has got reconnected.
The approach seems to fit your contraints and seems to be extensible. Only concern is the load on the main thread under very heavy load. The scalability may need to be tested here.
Second, I am sorry for my ignorance, but I fail to see any advantage that forking separate processes offers over spawing separate threads ... What you need to do with threads will have to be done with processes too ... It is just that you have additional IPC problems to deal with.
>Our customers will be running several clients (sometimes hundreds). Using the thread approach would require our
>customers to open more ports on their routers/firewalls and that isn't going to be possible or acceptable.
You can connect to one main thread and it can easily pass the socket to the server thread !!!
Third, I was hopeful of using SO_REUSEADDR in this scenario because this socket option would let another process bind to the socket and listen to the socket if no process is already listening to it. So you accept the new connection and pass it to server process who does all the send/recv operations on it ... I am not sure if this would work but it just may. That is the only way I can think of by which server process may be able to bind to a socket without creating it first.
Good luck
sunnycoder
ASKER
Hi sunnycoder,
Thanks for your response. I appreciate it. Unfortunately changing the architecture is just not going to be possible. The one running now has been in place for years and has been extensively tested and is extermely stable. Also, the application also must run on SCO Unix. More and more of our customers are switching to Linux but the app will still need to support SCO for some time and therefore the need for forking. What I'm trying to do is just add a bit of icing for linux customers that run our client/server app with wireless laptops so that when they go out of range they don't have to restart the app and can continue where they left off when they come back into range. There is also a vpn disconnect issue that some customers are having depending on their setup that this disconnect solution is going to try to address. I have implemented what I have discussed above and it's in the testing phase now. It is working well so far albeit a bit complex. Thanks again for taking the time and providing your comments. I am still going to effort a way to get away without needing that extra port but I'm not sure how much luck I will have. Thanks again!
- wdb
Thanks for your response. I appreciate it. Unfortunately changing the architecture is just not going to be possible. The one running now has been in place for years and has been extensively tested and is extermely stable. Also, the application also must run on SCO Unix. More and more of our customers are switching to Linux but the app will still need to support SCO for some time and therefore the need for forking. What I'm trying to do is just add a bit of icing for linux customers that run our client/server app with wireless laptops so that when they go out of range they don't have to restart the app and can continue where they left off when they come back into range. There is also a vpn disconnect issue that some customers are having depending on their setup that this disconnect solution is going to try to address. I have implemented what I have discussed above and it's in the testing phase now. It is working well so far albeit a bit complex. Thanks again for taking the time and providing your comments. I am still going to effort a way to get away without needing that extra port but I'm not sure how much luck I will have. Thanks again!
- wdb
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Tricky one!
It will not be enough to simply duplicate a socket. The problem is to determine how much data both sides have processed, and that a second invocation of the server's service results in identical data.
The only clean solution to this problem is that upon first connection by a client the server returns a session ID. The server then continously tracks at which position within a transaction the client (or, rather the session) is.
Once the connection is lost and regained, client and server can resume using the existing session ID.
This should be done on a separate logical layer, so your client/server does not see the interruption.
(For most applications, this is not quite simple to implement at application level. You need to be able to rollback a partial transaction on both client and server once the connection is lost.)
Cheers,
Stefan