Alright I have written a C# program that spiders the internet and I am having an issue where the spider is visiting the website more than once.
So I devised a plan to store the data in a flat text file which resulted in a horrible idea because of the volume of entries. I believe the next step is using a database.
I have decided to use MSSQL Compact Edition as opposed to MySQL or SQLite.
Now being a complete beginner with databases I am having trouble wrapping my head around how to make this work:
I have a table called "Projects" and it has two columns as of now, ID (Unique auto increment Key) and Name which is a user defined name. I would like to store ALL the websites the spider visits per unique project.
Project Key 1 name Blue Widgets this project spiders for the term blue widget and I would like to place all the sites it visits in the database and associate the two.