Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Questions about designing a data mining website and crawler

Posted on 2008-10-15
2
Medium Priority
?
780 Views
Last Modified: 2013-12-09
I have a few questions about a project I am working on. Being fairly new to the whole idea I decided to read up and found a great deal of information. The project has been designed and I have started writing the code for it but there are some issues that keep coming up.

Firstly, the bulk of of code comes in the form of class libraries that contain AI, rule processing and inference, database access, compression, etc. The crawler is also a class library that will reference the other libraries. The crawler will most likely be initialized by a console application or winform so that it will run outside of the asp.net session (any thoughts on running it from the asp.net website?).

So the first question is:
How can I control, manage, communicate with the web crawler when its running without using remoting or tcp client/server? Would I have to use a web service?

Second question is:
Is there a better approach to this design?

As it stands now I would like to have the crawler sit waiting for jobs to come in and then store the information into the database. I do not want the website to have to reference the libraries but still be able to access the data from the crawler and manage it as well.

My main concern is that if I use the scheduler I wrote to schedule the jobs and start the crawler the crawler will close out when the session from the site has ended. I am sort of lost on how to continue with this part.

I appreciate any help I can get and if I am being too vague just let me know and I will try to explain it in more detail and/or provide code snippets. Just as a side note, I am running SQL Server 2008, Windows Server 208 (IIS 7) and .NET 3.5 (Using Visual Studio 2008 to write it)

Thanks
Joe Wood
0
Comment
Question by:JoeDW
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 5

Accepted Solution

by:
wickedpassion earned 2000 total points
ID: 22739193
0
 
LVL 1

Author Comment

by:JoeDW
ID: 22749598
Wow, I really like the first link. It had tons of good information and I am sure it will keep me busy for a while. Thanks!!
0

Featured Post

Will your db performance match your db growth?

In Percona’s white paper “Performance at Scale: Keeping Your Database on Its Toes,” we take a high-level approach to what you need to think about when planning for database scalability.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

FAQ pages provide a simple way for you to supply and for customers to find answers to the most common questions about your company. Here are six reasons why your company website should have a FAQ page
When the s#!t hits the fan, you don’t have time to look up who’s on call, draft emails, call collaborators, or send text messages. An instant chat window is definitely the way to go, especially one like HipChat. HipChat is a true business app. An…
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
The viewer will get a basic understanding of what section 508 compliance can entail, learn about skip navigation links, alt text, transcripts, and font size controls.
Suggested Courses

704 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question