handling large mysql results in php

we have a database of about a half a million products that we need to upload to google.  the problem is after we run the query its very slow to go through each row with mysql_fetch_object... it takes something like 30-40 seconds just to take the data from 300 products to save them in the file.  how could we do this more quickly?

Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Beverley PortlockCommented:
I'm not sure about what you are trying to do. Are you trying to "process" 300 records from 500,000 or are you trying to upload 500,000 records via PHP to something at Google?

If you are trying to process records then I have two things that spring to mind.

1. Why read 500,000 if you only need 300?
2. If performance is an issue, have you ensured that every column mentioned in a WHERE, JOIN or ORDER BY clause has an index?

Sorry to be vague. If you could clarify your task some more then I could make a better go of it.
hillelbenAuthor Commented:
we need alll 500,000 - need to output the relevant data to a file d upload.  it just that at the rate its going now it would take over a day because mysql_fetch_object is very slow on such a large result set
Beverley PortlockCommented:
Have you looked at bypassing PHP and using SELECT .... INTO OUTFILE syntax?

Beverley PortlockCommented:
Here is an example http://www.mysqlfaqs.net/mysql-faqs/Data-Back-Up/Export-Data/How-to-use-SELECT-INTO-OUTFILE-statement-to-export-data

and always use a fully qualified path name for the result file.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
hillelbenAuthor Commented:
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.