MSSQL CE Keep track of URLs

Posted on 2011-05-10
Last Modified: 2012-08-14
Alright I have written a C# program that spiders the internet and I am having an issue where the spider is visiting the website more than once.

So I devised a plan to store the data in a flat text file which resulted in a horrible idea because of the volume of entries. I believe the next step is using a database.

I have decided to use MSSQL Compact Edition as opposed to MySQL or SQLite.

Now being a complete beginner with databases I am having trouble wrapping my head around how to make this work:

I have a table called "Projects" and it has two columns as of now, ID (Unique auto increment Key) and Name which is a user defined name. I would like to store ALL the websites the spider visits per unique project.


Project Key 1 name Blue Widgets this project spiders for the term blue widget and I would like to place all the sites it visits in the database and associate the two.

Any ideas?

Question by:snow4dayz

    Author Comment

    Furthermore I would like to create a hash with the url using CRC8 for quicker lookups. So save the URL to DB associate it with the unique project and create a hash that goes with the URL.
    LVL 27

    Expert Comment

    Here are some leads for you that talk about connecting to a sql database from

    Go ahead and start with something. Once you get stuck, post the actual issue and you'll get many people to help you rather than ask for 'how to do an entire project'.


    Author Comment

    I don't think I explained the question properly:

    Stuck on:
    How would I make a column accept hundreds of websites and URL CRCs? Such as: BJBGBVJKBNK HJGFHJBGVFHJF

    Again completely not understanding how to setup the tables, I know how to connect and read from them.
    LVL 3

    Expert Comment

    Hello, Just make the datatype of that column to NVARCHAR(MAX). This datatype will accept unicode values, so you can insert the URL.

    Accepted Solution

    After some intense study into MSSQL I found what I was missing there is a way to make tables relational in the VS2010 IDE, just was over looking it. Thanks for the input.

    Author Closing Comment

    Had to answer my own question, next time around I will better phrase my question. Thanks.

    Featured Post

    Do You Know the 4 Main Threat Actor Types?

    Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

    Join & Write a Comment

    Let's review the features of new SQL Server 2012 (Denali CTP3). It listed as below: PERCENT_RANK(): PERCENT_RANK() function will returns the percentage value of rank of the values among its group. PERCENT_RANK() function value always in be…
    Load balancing is the method of dividing the total amount of work performed by one computer between two or more computers. Its aim is to get more work done in the same amount of time, ensuring that all the users get served faster.
    Via a live example, show how to extract information from SQL Server on Database, Connection and Server properties
    Viewers will learn how to use the SELECT statement in SQL to return specific rows and columns, with various degrees of sorting and limits in place.

    734 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    17 Experts available now in Live!

    Get 1:1 Help Now