Improve company productivity with a Business Account.Sign Up


.Net: Parsing websites and storing in SQL Server

Posted on 2004-03-28
Medium Priority
Last Modified: 2010-04-17
Hi, I'm new to windows programming and I need to do a program running on server, downloading some pages from web, parsing and storing in a SQL-server. The program should be performing it's tasks on scheduled times.

Now, what I'd like to ask is some general guidance... What kind of program do you suggest I do?

 I have most of the code ready in C# .Net application (windows forms) which could be run on selected intervals by Windows scheduler (or I could do a timer in the app itself?), but is there some kind of server application or similar (without the UI?)? I'm looking for a way that doesn't eat too much performance....

Thanks in Advance,
Question by:mmiika
  • 2

Accepted Solution

eric_duncan earned 750 total points
ID: 10701033
Here are two suggestions to meet your needs:

First, you could create a Windows Service (this website has a lot of information about this: To do this, you would have to use a timer inside of the service itself because the service will be running continuously in the background and, thus, can't be scheduled using the Windows Scheduler.

Alternately, you could create a simple Console application that is scheduled using the Windows Scheduler. This is far easier to implement than the service, but if you REALLY need to make sure the application fires at every specified interval, the Windows Scheduler may not be reliable enough for you, particularly if the machine is used heavily. Believe it or not, Windows Scheduler does not guarantee that a particular task will be executed at a given time - instead, it just guarantees that the task won't be executed BEFORE the specified time. It will TRY to run it as soon after the specified time as possible, but that doesn't mean that it WILL run it. If you want to go this route, there are some cron utilities out there for Windows that are far more reliable than the Windows Scheduler that you could use the schedule the application.

The question of which of the above solutions is superior comes down to a question of personal taste. The service will use slightly more CPU cycles than the console application, but if you follow my advice and install a 3rd party cron utility for scheduling it will run as a service and so will increase the overall CPU footprint of the console app such that it will meet or exceed that of the service. At any rate, the performance hit we're looking at is so minimal that it probably isn't worth worrying about it.

Let me know if this helps!


Expert Comment

ID: 10706616

I work for a company by the name of Crocus Plains Software Ltd, located in Calgary, AB, Canada.   We have been working on a project that does exactly what you are looking for, and are looking for more Beta testers.   The scheduler runs across a network, and as the parsing of these websites can be extremely CPU intensive, the scheduler manages as many application servers that you need, which run the actual parsing.  

If you are interested in becoming a Beta tester, please contact us.
(note: currently the website for this project does not have information)

Brett Swift

Expert Comment

ID: 10722713
Thanks for the points! Let me know if you have any more questions.


Author Comment

ID: 10729062
Sure, I did it as console app... might test doing a service later, just afraid that there's more problem getting it work.

Thanks, Miika

Featured Post

What Kind of Coding Program is Right for You?

There are many ways to learn to code these days. From coding bootcamps like Flatiron School to online courses to totally free beginner resources. The best way to learn to code depends on many factors, but the most important one is you. See what course is best for you.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

In real business world data are crucial and sometimes data are shared among different information systems. Hence, an agreeable file transfer protocol need to be established.
This article describes and provides a custom-made tool I wrote to give businesses a means of identifying commercial music content, without having to expend too much effort. Business recordings are easily identified from possibly illegal music files …
Introduction to Processes

601 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question