I would like to place 20 million URLs (10 gig) in a mySQL database.
I have the URLs in a text document in this format:
URL | Title | Description | Category
I will use LOAD DATA INFILE to place these in the database.
I have been told that MySQL should be able to handle this amount of data if the database is properly indexed.
Here is the query that will be run:
SELECT `URL`, `Category` FROM `pages` WHERE MATCH (`Category`, `Title`, `Description`) AGAINST ('Hurricane' WITH QUERY EXPANSION);
I want to maximize performance. How should I setup the database? And what field(s) should be indexed? Should I have a unique number id for each row? What should be the primary field? Would it be best for me to place the category names in a separate table and then only address the ID of that category in my main row?
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
Hi Hank,
As a matter of practice, I almost always apply an integer primary key to a table. It makes certain things a lot easier when the application changes at a future date.
But no, unless you need to guarantee uniqueness or have a need to structure the data on disk in a particular manner you don't need a primary key.
Kent
I ought to warn you that I'm *guessing* that there will be a performance benefit indexing `Category` for SELECT UNIQUE `Category`...
I haven't confirmed this is true, but it makes sense.
Indexing columns has a cost implication when you insert records, so you might want to test this assumption to confirm that there is a performance benefit for the SELECT UNIQUE.
It definitely is the case that you want to FULLTEXT (`Category`, `Title`, `Description`) for the sake of the fulltext search.
[Agreed with Kent about the 1y key.]
Business Accounts
Answer for Membership
by: KdoPosted on 2009-08-24 at 05:54:39ID: 25167778
Hi Hank,
Before you build this table you need to know how it will be queried. If one of the queries that you will support is "what are all of the .com addresses?" you'll want to split the URL into "words" or "domains" so that the query can be properly indexed. Searching for text within the URL string (or at the end) is inherently inefficient. You'll want to develop a means to improve it.
Kent