[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Indy Client Server freezing

Posted on 2009-04-17
24
Medium Priority
?
3,916 Views
Last Modified: 2013-11-23
Hi experts.

I am having a really hard time to solve an Indy10  communication problem. I have a client-server project which uses TIdTCPClient and TIdTCPServer to send files across the network (LAN). The application must be able to send files in both directions: client to server and server to client. At the moment I am testing server to client part. I send files using fixed length chunks.

Sometimes the communication loop will freeze at a random iteration in the line where the client confirms  reception of the chunk (it writes 1 byte response to the handler) . For testing I am using Windows XP SP3 file (313 MB) because with large files the problem is more likely to happen.

Some important facts you must know before answering:

- I have downloaded latest version of indy 10 available from SVN, few weeks ago.
- Server and Client sending loops are out from main VCL thread.
- I have IdAntifreeze components on both sides
- I have tried differents types of reading/writing to the IOHandler: ReadByte, ReadStream, ReadLn...
- I use MadExcept and exhaustive logging to find out things, so I am quite sure that the read instruction in the server does not exit, but the client does write the response byte.


Does anyone know a reason why the TIdTCPServer does not receive the client response while the TidTCPClient skips to the instruction after the write? They are suposed to use blocking sockets... It does not make any sense to me.

Any help will be greatly appreciated.

PS: I may not be able to read this post until monday, but i just could not wait.
0
Comment
Question by:praxalia
  • 13
  • 5
  • 2
  • +2
23 Comments
 
LVL 18

Expert Comment

by:Johnjces
ID: 24170552
One thing you might try is using two different ports instead of one. Server receives on port 10001 and sends on 10002 and clients send on 10002 and receive on 10001.

"Sometimes" I have seen that transmission of TCP/IP packets on the same port number has caused some conflicts.

Might help, might not, but worth a try.

John
0
 
LVL 13

Expert Comment

by:ThievingSix
ID: 24173927
I would highly recommend moving back to Indy 9 for all around stability. When using Indy 10 I have come across some problems that switching back never had.
0
 

Author Comment

by:praxalia
ID: 24174913
Hi Johnjces.

Can server and client use two different ports for sending and receiving? Didn't know that, I thought each connection was binded to one specific port. Could you please send me an example? Thanks.
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:praxalia
ID: 24174945
Hey ThievingSix, thanks for your comment.

Yes, in fact I have thought about going back to Indy 9 times, but I 'd rather prefer to stay in indy 10 as long as possible. I may want to ensure unicode compliance to my project later on or even move it to Delphi 2009 (now I am using Delphi 2006). In my opinion, going back to indy 9 (which I think is being barely maintained right now) could throw me into more problems in the short term.
0
 
LVL 18

Expert Comment

by:Johnjces
ID: 24176184
Try it! You have separate client and server components and it can work.
0
 

Author Comment

by:praxalia
ID: 24178449
Hi Johnjces, I talked about using the same TIdTCPServer and TIdTCPClient, of course I know they are separate components ;).

The point is that I do not want to use more than one TIdTCPClient and TIdTCPServer in each module, that would be the last option. I need to keep the project as simple as possible, experience has showed me that debugging multithreaded projects can be absolutely discouraging and time consuming. Two different connections on both sides means lots of extra code: paired connection and disconnection events, two different timers to reconnect sockets in case of failure, two synchronized server threads for each client connection besides the main VCL thread... There's too much code attached to one single connection to think in handle two of them per client, at least at the moment. If there would be a way of using two ports in one connection that would be fine, but I understand that this is not possible, by definition. Thanks anyway for your suggestion.

What I really need to know is why does a blocking socket get data in the client side which does not come out in the server side.

I am working on a simplified project which will show the problem. Tomorrow I will have it ready. If you need the code, just ask me.




0
 

Author Comment

by:praxalia
ID: 24196312
Hi experts. following the principle: no news = bad news, I have just finsihed a simple client-server version where the freezing arises. The problem is that EE won't allow me to upload the zip archive with the project files, and even though the project is small, the code is too long to fit as a comment.

How can I publish the code in a civilized manner? Didn't find the answer in EE help or FAQs. Thanks
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 24222947
use http://www.ee-stuff.com
you can put EE attachments there, and it will make a link that you can paste here
I do it all the time
If you are unable to do that, then email it to me (my address is in my profile) and I will put it on EE stuff for you, for all to see. This should still be within the EE rules.

As for your problem, I have various demos that do what you are asking, but we will modify your code if that is what you prefer.
FWIW, sending the "length of the stream", followed by the stream, is the way to go (imho)

Even though indy uses blocking sockets, TCP is still a slightly random beast
hardware and the like can change things...
not like UDP, but for example, you may send 1 "packet", but it may arrive as several packets, and even have large delays between packets.
Even though Indy 10 is a big change from Indy 9, I can only re-iterate what Chad Z Hower (author of Indy) told me.
I was having problems with Indy 10, and he suggested I download the latest "development snapshot", not the official release. This was a while ago now, but it certainly contained many fixes to the official release.
Give that a go.
Looking forward to seeing your code...

oh, and please tell me what version of Delphi you are using. That will make a big difference to us.

0
 

Author Comment

by:praxalia
ID: 24222954
Hi experts. I just uploaded the project files so you can test them and see the problem. I have tried to make it easy to test: connection can be done on any port (it defaults to 9997), and to any host name, without having to retype the code. I have translated screen messages and made an effort to remove unimportant things. You should try to send a big file over the lan (I use a 300 Mb file), because it may not freeze with small ones.

https://filedb.experts-exchange.com/incoming/ee-stuff/7612-project.zip

Thanks
0
 

Author Comment

by:praxalia
ID: 24223047
Hi TheRealLoki, thanks for your tips. I received an administrative comment telling me how to upload files to www.ee-stuff.com, so I have just uploaded them. Thanks for your offer, anyway.

As you say, I would rather prefer you having a look at my code. It could be that the origin of my problem was in a particular way I am using threads or indy components, so I prefer to get solutions which will apply straight to my project.

I do use the "length of stream followed by the stream", approach, to send the chinks. In fact I have tried several different ways, but this is the one that remains.

About having the latest snapshot available, I did install tortoise svn client to download latest snapshot of indy 10 available on the trunk folder (following instructions found in indy project's page). This was three weeks ago, more or less. If you can run the project without freezing, I assure you that I will download the latest snapshot straight away.

I am using Delphi 2006 Professional, BTW.
0
 

Author Comment

by:praxalia
ID: 24248468
Hi there, TheRealLoki. While you were looking at the code I downloaded the latest snapshot from Indy 10 and installed it (after uninstalling current version completely). The problem remains the same, it still freezes. If you cannot find the time to look at my code, please send me one of the working examples you mentioned above of sending files in chunks using indy10, so I can compare them with my project. I will add differences one by one to see what is that makes it crash. I am really stucked now with this issue. Thanks.
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 24257364
sorry, I've been off work, so havent checked EE.
It's probably a thread issue, but I''ll see if I can reproduce it on my home dev pc
I'm concerned about the way you use the TIdContext.
Give me a few hours to try and get my dev pc up :-)
0
 
LVL 17

Accepted Solution

by:
TheRealLoki earned 2000 total points
ID: 24266203
Your code locks up here also
My demos are up on
http://sourceforge.net/projects/internetdemos
use any of the indy 10 demos, and look for CMD_FILE or TFileStream

I generally send a "FILE" command followed by the filename and the size
then the actual data

0
 

Author Comment

by:praxalia
ID: 24268498
Hi  Loki,

I downloaded Indy 10 client and server and I'm trying to compile them. In both server and client I get some strange "Incompatbile types integer and Int64" compiler errors, in the onConnect event, where you assign OnWork, and OnWorkBegin connection events. I am looking at it at the moment.

When I get to compile your projects I will try to apply differences with my project, one by one, so to see which one will lock things up. First things I see different:

- You create the TIdTCPclient at runtime
- You send the file in a single write instruction.
- TCP dialog is started by a client's command
- You use Synchronize in the client, to call VCL functions from the thread (I use windows messaging, like you do in the server).

Maybe I left someone out, it's just a first inspection.

I will tell you how I get on with this. If you discover the origin of the freezing problem meanwhile, please tell me.

Thanks.
0
 

Author Comment

by:praxalia
ID: 24303229
Hi Loki. First of all, my apologies, i could not work on your demos until this morning. We had a 4 day weekend here.

It looks like I am close to isolating the problem, now.
Using your freezing-free client/server demo as a basis, I began to add changes pasting pieces of my code into your project.

1) Made server send the file to the client in the OnConnect event: Worked OK
2) Sent the file in chunks: OK
3) Added a button in the server to send the file at any time, using a custom created thread: FREEZED!!!

So, the problem arises when I try to use my own custom thread to send the file from the server, instead of using the thread spawned by the TIdTCPServer. The problem is that, as sending the file to the client will be the user's decission, I must be able to send the file at any time and I cannot do it without spawning a thread outside indy's. Also, I do not want to freeze users interface with lengthy file operations, so I cannot use the main VCL thread either.

The custom thread is the most simple thread I can imagine, I just create it, resume, and it does not even have a execution loop (I wanted to make it as low CPU time consuming as possible), just sends the file and terminates. The code to send the file is the same thread-safe procedure I use from the onconnect event.

I have uploaded the project to ee-stuff. You will see a checkbox and a button. With the checkbox enabled, it sends the file as soon as the client connects using indy thread (works OK). If you use the button, it creates a new thread and sends the file (Freezes).

If you see where is the problem, tell me. I am working in it right now.

Thanks.
0
 

Author Comment

by:praxalia
ID: 24316693
I have been trying around things and this is what I have seen:

1) If I use threads spawned by TIdTCPServer in the OnConnect or the OnExecute events, the file is sent OK. If I use OnExecute, I can activate a boolean flag in the main form when I click a button and test it inside the event. This is a partial solution because I can respond to a user action to send the file.
2) Whenever I use a custom thread it might interfere with indy's own listener threads and at some point my thread gets blocked.

Using the onexecute thread implies that the onexecute event won't be triggered again for any other client while the file is being sent? This would be an important drawback, as the server can have several connected clients, all of them blocked until the operation with a single client is finished.

I think that the neatest solution would be using my own thread, but at this point I don't know what else I can do. I already tried different thread's priorities, changed the way I store the IdContext, tested TIdThread instead of TThread... with no results.

Any ideas?
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 24329638
Using the onexecute thread implies that the onexecute event won't be triggered again for any other client while the file is being sent? This would be an important drawback, as the server can have several connected clients, all of them blocked until the operation with a single client is finished.
This is incorrect. Indy's Execute event fires for EACH client connection. so you may in fact have multiple Execute threads running at the same time.
0
 

Author Comment

by:praxalia
ID: 24394242
Sorry for the delay. I have been testing the demos...

First of all, you're absolutely right about OnExecute triggering. It triggers for each client independently, I made a bad reading of documentation. This helps me greatly because I can use Indy's own threads to control de TCP flow in a safe manner. I guess I will have to retype the code to move all TCP commands inside the OnExecute thread in the main form. I tested PostThreadMessage to notify server's thread about starting operations, and it works fine.

It would be great if I could just understand why using a different thread to access TIdTCPServer connection ends up in unpredictably freezing. I made it all really simple: I protect socket access with critical sections and it should be all thread-safe! Could it be a bug somewhere in indy's thread management? I couldn't find any example in the web which used a custom thread to read/write to a server. I even found a post somewhere that recommended (without further explanations) to use indy connections always from the same thread spawned in the OnConnect event, even though synchronizing objects could be used to protect access from other threads.

If no one comes with an explanation to this problem I will award you the points. Your demo code was useful to clarify things.

0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 24405384
Indy uses a TidThread which has a few extra methods. I expect using your own TThread means that you can't inherently call some of the TiDThread's OnExecute code which probably handles some other syncing that isn't obvious before the published Execute method.
e.g.
Indy "execute"
  "do indy stuff"
  if assigned(fOnExecute) then fOnExecute(self, AThread);
  "do other indy stuff"
end;

I still do a lot of guess work when it comes to Indy

I have done inter thread communication on Indy using TCriticalSection, but it probably wouldn't have been a good match for your situation.
0
 

Author Comment

by:praxalia
ID: 24411763
Yes, that makes sense. I even tried with my own TIdThread, just in case the origin of the problem was on synchronizing between a TThread and a TIdThread, but it didn't work either.

As I said, I award you the point for your response with the demo. Thanks for your help, guys.
0
 

Expert Comment

by:DKropf3
ID: 25990745
Hi Praxalia,

i think I have the same problem as you did.
Did you ever solved this?

Thanks!
0
 

Author Comment

by:praxalia
ID: 25999492
Hi DKropf.

Yes I did solve the freezing issue. What I discovered is that, for some reason, TIdTCPServer socket is only safe  to use inside the Onexecute thread of the TIdTCPServer. My problem arosed from the fact that I called socket operations (read and write) from my own custom thread. The program freezed randomly at these I/O operations no matter which blocking system I used (semaphores, Critical sections, signaled events...) to avoid conflicts with Indy's own thread. At last, everything worked fine when I moved all reading/writing to the socket inside the OnExecute loop.

Hope it helps,

praxalia

0
 

Expert Comment

by:DKropf3
ID: 26002285
Hi Praxalia,

thanks for answer, but i solved it to. :)

My problem was bizarre! Calling .Connected from different threads without some blocking mechanism will interfere with buffers and on some ocassions add trash to buffer ... Spent a month searching ... :/

Thanks anyway!
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Programmer's Notepad is, one of the best free text editing tools available, simply because the developers appear to have second-guessed every weird problem or issue a programmer is likely to run into. One of these problems is selecting and deleti…
Jaspersoft Studio is a plugin for Eclipse that lets you create reports from a datasource.  In this article, we'll go over creating a report from a default template and setting up a datasource that connects to your database.
The viewer will learn how to use NetBeans IDE 8.0 for Windows to connect to a MySQL database. Open Services Panel: Create a new connection using New Connection Wizard: Create a test database called eetutorial: Create a new test tabel called ee…
The viewer will learn how to use and create new code templates in NetBeans IDE 8.0 for Windows.
Suggested Courses
Course of the Month18 days, 19 hours left to enroll

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question