.Net: Parsing websites and storing in SQL Server

Posted on 2004-03-28
Last Modified: 2010-04-17
Hi, I'm new to windows programming and I need to do a program running on server, downloading some pages from web, parsing and storing in a SQL-server. The program should be performing it's tasks on scheduled times.

Now, what I'd like to ask is some general guidance... What kind of program do you suggest I do?

 I have most of the code ready in C# .Net application (windows forms) which could be run on selected intervals by Windows scheduler (or I could do a timer in the app itself?), but is there some kind of server application or similar (without the UI?)? I'm looking for a way that doesn't eat too much performance....

Thanks in Advance,
Question by:mmiika
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2

Accepted Solution

eric_duncan earned 250 total points
ID: 10701033
Here are two suggestions to meet your needs:

First, you could create a Windows Service (this website has a lot of information about this: To do this, you would have to use a timer inside of the service itself because the service will be running continuously in the background and, thus, can't be scheduled using the Windows Scheduler.

Alternately, you could create a simple Console application that is scheduled using the Windows Scheduler. This is far easier to implement than the service, but if you REALLY need to make sure the application fires at every specified interval, the Windows Scheduler may not be reliable enough for you, particularly if the machine is used heavily. Believe it or not, Windows Scheduler does not guarantee that a particular task will be executed at a given time - instead, it just guarantees that the task won't be executed BEFORE the specified time. It will TRY to run it as soon after the specified time as possible, but that doesn't mean that it WILL run it. If you want to go this route, there are some cron utilities out there for Windows that are far more reliable than the Windows Scheduler that you could use the schedule the application.

The question of which of the above solutions is superior comes down to a question of personal taste. The service will use slightly more CPU cycles than the console application, but if you follow my advice and install a 3rd party cron utility for scheduling it will run as a service and so will increase the overall CPU footprint of the console app such that it will meet or exceed that of the service. At any rate, the performance hit we're looking at is so minimal that it probably isn't worth worrying about it.

Let me know if this helps!


Expert Comment

ID: 10706616

I work for a company by the name of Crocus Plains Software Ltd, located in Calgary, AB, Canada.   We have been working on a project that does exactly what you are looking for, and are looking for more Beta testers.   The scheduler runs across a network, and as the parsing of these websites can be extremely CPU intensive, the scheduler manages as many application servers that you need, which run the actual parsing.  

If you are interested in becoming a Beta tester, please contact us.
(note: currently the website for this project does not have information)

Brett Swift

Expert Comment

ID: 10722713
Thanks for the points! Let me know if you have any more questions.


Author Comment

ID: 10729062
Sure, I did it as console app... might test doing a service later, just afraid that there's more problem getting it work.

Thanks, Miika

Featured Post

The Ultimate Checklist to Optimize Your Website

Websites are getting bigger and complicated by the day. Video, images, custom fonts are all great for showcasing your product/service. But the price to pay in terms of reduced page load times and ultimately, decreased sales, can lead to some difficult decisions about what to cut.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you’re thinking to yourself “That description sounds a lot like two people doing the work that one could accomplish,” you’re not alone.
This article will inform Clients about common and important expectations from the freelancers (Experts) who are looking at your Gig.
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …
With the power of JIRA, there's an unlimited number of ways you can customize it, use it and benefit from it. With that in mind, there's bound to be things that I wasn't able to cover in this course. With this summary we'll look at some places to go…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question