.Net: Parsing websites and storing in SQL Server

Posted on 2004-03-28
Last Modified: 2010-04-17
Hi, I'm new to windows programming and I need to do a program running on server, downloading some pages from web, parsing and storing in a SQL-server. The program should be performing it's tasks on scheduled times.

Now, what I'd like to ask is some general guidance... What kind of program do you suggest I do?

 I have most of the code ready in C# .Net application (windows forms) which could be run on selected intervals by Windows scheduler (or I could do a timer in the app itself?), but is there some kind of server application or similar (without the UI?)? I'm looking for a way that doesn't eat too much performance....

Thanks in Advance,
Question by:mmiika
  • 2

Accepted Solution

eric_duncan earned 250 total points
ID: 10701033
Here are two suggestions to meet your needs:

First, you could create a Windows Service (this website has a lot of information about this: To do this, you would have to use a timer inside of the service itself because the service will be running continuously in the background and, thus, can't be scheduled using the Windows Scheduler.

Alternately, you could create a simple Console application that is scheduled using the Windows Scheduler. This is far easier to implement than the service, but if you REALLY need to make sure the application fires at every specified interval, the Windows Scheduler may not be reliable enough for you, particularly if the machine is used heavily. Believe it or not, Windows Scheduler does not guarantee that a particular task will be executed at a given time - instead, it just guarantees that the task won't be executed BEFORE the specified time. It will TRY to run it as soon after the specified time as possible, but that doesn't mean that it WILL run it. If you want to go this route, there are some cron utilities out there for Windows that are far more reliable than the Windows Scheduler that you could use the schedule the application.

The question of which of the above solutions is superior comes down to a question of personal taste. The service will use slightly more CPU cycles than the console application, but if you follow my advice and install a 3rd party cron utility for scheduling it will run as a service and so will increase the overall CPU footprint of the console app such that it will meet or exceed that of the service. At any rate, the performance hit we're looking at is so minimal that it probably isn't worth worrying about it.

Let me know if this helps!


Expert Comment

ID: 10706616

I work for a company by the name of Crocus Plains Software Ltd, located in Calgary, AB, Canada.   We have been working on a project that does exactly what you are looking for, and are looking for more Beta testers.   The scheduler runs across a network, and as the parsing of these websites can be extremely CPU intensive, the scheduler manages as many application servers that you need, which run the actual parsing.  

If you are interested in becoming a Beta tester, please contact us.
(note: currently the website for this project does not have information)

Brett Swift

Expert Comment

ID: 10722713
Thanks for the points! Let me know if you have any more questions.


Author Comment

ID: 10729062
Sure, I did it as console app... might test doing a service later, just afraid that there's more problem getting it work.

Thanks, Miika

Featured Post

Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This is an explanation of a simple data model to help parse a JSON feed
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question