Delphi 2010 Seperate Exe vs Threads

I have an application developed in Delphi 2010, running on a networked PC which has to scan a large number of backup files over servers and check the modify dates. If a back up file's date changed, it must test the file to see if its corrupt, and if its OK, copy it to a common folder on one server, Task1.  The newly copied file then needs to be further processed in Task 2 which involves an intense process and then updating the processed information in an SQL database.  Then there is the user interface which enables a user to interact and analyse results from the SQL database, Task 3.

At the moment I have the three tasks running in a single program with Task 1 and 2 coordinated by a self made scheduler and regularly calling Application.ProcessMessages to give the user  a chance to interact with the program.  This works, but not very smooth.

I am thus contemplating moving the three tasks into separate threads, but I am really battling to understand the working of threads.  I played with some examples, but I still can't get my own Procedures and Functions to process inside the new thread.  My alternative thinking was to move the three tasks into 3 separate executables.

I would guess the thread method would be technically the right thing to do and synchronisation won't be a problem since each of the tasks just need to talk to the SQL database in them self.  I thus need some sound advice as to what would be the best route and if it’s the thread route, how can I find out how these really work and how to implement it correctly.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Emmanuel PASQUIERFreelance Project ManagerCommented:
Then there is the user interface which enables a user to interact and analyse results from the SQL database, Task 3.
That part is considerably different from other 2, in the sense that it is a User application.
If, by "enables a user to interact" you don't mean interacting with the other 2 tasks, but only interact with the DB data resulting from these tasks, then obviously you SHOULD put all these functionalities in a separate application. You won't need then to use tricks such as too much Application.ProcessMessages to make it smooth for the user.

That leaves the other 2 tasks, that in my opinion could well be placed in 2 threads of the same application, or in two separate applications. That would not change anything performance-wise. The real answer for this choice would depend on what user-level interaction remains for only those 2 tasks, and how these tasks are supposed to work : continuously detecting files modifications (that can be costly in a network environment) or it would be a regular scanning application that does the detection and process once, then quit (so, a 'console' application without GUI, that can be scheduled with a task scheduler)

If you don't have specific constraints needing those tasks to communicate beyond the fact that files will be moved or modified, I would consider making 3 applications :
1) console application that scans a specified network folder for modification and copy the files
2) console application that scans the target folder and process files
3) GUI application for all user interaction with results

Note : console applications can be mounted as service with a little MS tool : srvany
That does not require any modification to the application (that could still be launched in commande line)
That is by far the easiest and the most versatile way to create services with Delphi
HenryM2Author Commented:
epasquier, thank you for your answer.  I should have mentioned that from a user perspective, "solution" should appear to be a single application, with also limited user interaction with Task 1 and Task 2.  These at the monent live in separate Forms in the application and is used for making setting changes to the two scan tasks and to check progress when required by the user.

In a way, I am releved that you seem pro, separate applications as that would probably be the easyes from a programming point of view for me to implement, but then i need to considder what I say above.  Will the console application mentod solve this, I am not experience in creating and using console applications?
Emmanuel PASQUIERFreelance Project ManagerCommented:
The advantage of separate the tasks in different applications is that you can distribute the work in different workstation, and put the 2 first tasks on the machines that would reduce or eliminate network trafic.
for task 1, it should run as a service in the same machine as the folder to monitor. That has the excellent advantage of being able to use OS notifications EVERY file change in a specified folder, which is a lot better than polling constantly a network folder for modifications. Use the unit below for that.
The filters of the actions notified should be FILE_ACTION_MODIFIED / FILE_NOTIFY_CHANGE_LAST_WRITE

for task 2, it shoud also run as a service, but in the machine hosting the destination folder of the 1st task.
You will again use the same DirMonitor unit, this time detecting the new files with FILE_ACTION_ADDED

If you do that the right way, you will eliminate the network bandwidth usage for those 2 tasks, and these tasks will do nothing (no CPU eating) until something happens to the folders they have to monitor

Note : to use the unit below, you have to install it with 'Component' menu, then 'Install Component'
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Emmanuel PASQUIERFreelance Project ManagerCommented:
you can always implement a TCP/IP communication between the different applications, to let the user set the parameters for both tasks in his GUI app, then when those settings are validated, you signal simply the console applications to reload the configuration, or to start/stop monitoring
All that your main GUI application has to know for that, is the IP address where the 2 task applications run.

As I said, the main argument AGAINST putting all this in the same application, is that since you move/read files a lot from different network folders, you can litterally shoot down your bandwidth if your tasks use a polling approach
Emmanuel PASQUIERFreelance Project ManagerCommented:
ah, by the way, creating 'console' applications with Delphi is even simpler than creating GUI application, you just have to say it to Delphi : File -> New -> Other then
Delphi Projects -> Console Application

If you need TCP/IP messaging between your applications, an even better way would be to create your console applications as WEBSERVICE (console), using again Delphi wizards to make the tricky work for you.
You would then expose a few WebServer methods like : ReloadConfig , StartMonitoring , StopMonitoring

Your GUI application would then be able to call these remote procedures almost as simply as if it was standard methods in the same process.
more documentation on this :
HenryM2Author Commented:
Thank you, I just need to digest all of this and try out a few things.
HenryM2Author Commented:
epasquier, if were to ignore the fact that the applications were to scan over different servers, for now, taking into account that Web Service and Console Applications are new to me, my thinking is as follows:
If I should have the GUI program from where the user can interrogate the Databese, and from the same GUI the user can set parameters for Task 1 and 2 which I think may work well as console applications, then will it also be possible for the Gui to display progress information received back from Console apllications task 1 and Task 2 ?
Emmanuel PASQUIERFreelance Project ManagerCommented:
well, yes I thought you would eventually come with this question, but decided to not make knots in your brain until you asked.
The solution, based on WebService as well, would be to make your GUI app ALSO a WebService (but a GUI one, in the Delphi wizard), with 2 functions : Task1Log and Task2Log, with a string parameter.
Then console application 1 and 2 (we will call them C1 & C2) can return logs to the GUI application, this time by being the clients of a webservice. Your GUI app will then only have to add those logs to a memo or listbox or listview, either the same for both tasks or in separate views. As you see fit.
It would probably means that you also will have to change a bit the methods exposed by C1 and C2, so that GUI app can tell them its IP for the calls of log functions.

I will admit that you could built such a system with simple socket components, but if you never did that then WebService is far simpler (after the initial reading and understanding of the articles I sent you)
Emmanuel PASQUIERFreelance Project ManagerCommented:
about the different servers to scan : well, you would simply add different such C1 and C2 applications on each servers, and make your GUI application able to follow more than one set of such applications (so, in terms of WebService, a few instances of the same HTTPRIO objects, using different IP parameters)
Emmanuel PASQUIERFreelance Project ManagerCommented:
You can also need 2 or more GUI apps running, on different workstation (by different users) monitoring different sets of servers. The only restriction is that only ONE GUI app will receive the progress/logs of each C1/C2 application (the last one that connected to it), except if you want to support multiple *backlog* GUI application per console workers.
I recommend against such complication.
WebService is the key to versatility and load balancing, at the price of lesser performance than direct socket system (but who cares), while allowing a very simple programmation of it all. You can concentrate on the more complex global picture of your system without worrying about the lesser communication details.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
HenryM2Author Commented:
I think I am getting the picture with a bit of unknotting to do around the use of Web services, this seem to be the way to go.  To recap, I can have the two Console Applications Tasks C1 and C2, runing on the server (I will scan from one server for starters and look at your DirMonitor App later) and then the users will run the WebService Gui app, mainly interrogating the SQL Databse.  I will in any way only give one of the users (supervisor) rights to adjust settings and interface directly with the Console Appication tasks, so the complexity of multiple *backlog* ? won't be necessary.

I just need a bit of time to try this out, so I will keep the question open, incase I get stuck.  Thanks so far.  
Emmanuel PASQUIERFreelance Project ManagerCommented:
you're welcome !

You can always close this as it gives the general principle, and I think you understood it. Then if you have specific difficulties, you can open new questions, otherwise probably no other experts will look at this thread later to answer.
HenryM2Author Commented:
OK, with most of the knots becoming undone, in the second document they state "Then in the Delphi IDE go to Projects, then options, and then set the output directory to "X:inetpubwwwrootcgi-bin." "X" is for whatever drive name your web server root is located on. " 

I am just running this on my development PC, offline, and I am not to sure that I have a web server installed.  Is it necessary to install a web server if I just want to use this to transfer messages between my GUI and the WebService applications C1 and C2.
HenryM2Author Commented:
I understand exactly how the proposed solution of Web Service Server - Web Client Client will solve my problem.  I will however need some practical guidance to get this working.  The question thus, has been answered very well.
Emmanuel PASQUIERFreelance Project ManagerCommented:
set the output directory to "X:inetpubwwwrootcgi-bin." "X" is for whatever drive name your web server root is located on. " 

arh, don't use WebServer extension (ISAPI DLL, or CGI), but standalone Webservice application using INDY TCP/IP components as the  communication channel. Otherwise, yes you will have to setup a web server.
When you run Delphi Wizard for a new SOAP server application (in Delphi\Web Services templates), you select 'Indy VCL application' for the GUI app, and 'Indy console application' for C1 and C2
HenryM2Author Commented:
Ohh choices choices choices.  On my system I don't seem to find 'Indy VCL application' for the GUI app, and 'Indy console application'

I have
                   Soap Server Application | SOAP Server Data Module | SOAP Server Interface | WSDL Importer

None of above offer the Indy option.
I closed the question as you sugested, so we can continue under my new post, asking for an example.  
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.