How to Bulk import 20,000 products into magento fast

Hi Fibo,

Thank you for the quick reply.

I understand the increased bucket size thanks for the tip.

In giving the script as much ram as possible do I have to change any settings in the Magento files or in php.ini on my vps?

This will be an infrequent update i.e the product database will refreshed every 4 months.

"inhibit table catalogs" does this mean disable flat catalog in magento confirguration, and are you recommending i disable cache too?

Hi,
Glad I could help, thx for thee grade and points.

- If your Magento app is running alone on thhe server, give it almost all the ram you have available; all the time. If it is not, just do that when you are running the update

- The simples place to change the ram size is probably php.ini BUT check the .htaccess that comes with Magento which has also some limitations.

- Yes, disable the flat catalog for the import. Fort the cache: as far as I remember the import will go faster with cache disabled, but I wouls suggest you check on some test runs.

hmm  I don't see the ,htaccess file in my route directory

There is probably an .htaccess file in your magento directory as well as in the /app subdir.
However, depending on your ftp client setup, these .xxx files might not be visible.

Which ftp client are you using?
Is there some option that you can change? (eg in Filezilla "force hidden files display")

sorry fibo, found th directory last night was using file zilla.

What I ahve done + results

Magento admin interface
***********************
Navigated to - System> Cache Management
Disabled All cache
Saved Settings

Navigated to - System> Configuration > Catalog
Use Flat Catalog set to "NO"
Saved Settings

Magento .htaccess
***********************
memory_limit = 1024M

VPS settings PHP.ini
***********************
memory_limit =1024M

Restart VPS

Magento backend
***********************

Magento backend > System > Profiles > My Profile
***********************
bucket size set to 500

Tested import of 500 products

Result

1 product per second

So approximately 50 minutes to complete this import

I have 20,000 products to import with 50 fileds that make-up my .csv import file, going in to 2000 categories.  While this was a huge improvement on my previous time of 8-20 seconds per product. I still need a faster way.

Any ideas?

Glad that you were able to reach 1 product/ second, this is indeed a nicely quantifiable improvement.

I presume you are importing from a file you have uploaded to the server, so that no processing is consumed by that.

bucket size 1000?
The improvement will not be that great though: you can assume that processing time for a bucket of size n is made of
bucket overhead + n times processing of 1 record.
or for 1 record:  1processing + (1/n) bucket overhead

So going from 500 to 1000 would make you gain per record (1/500) - (1/1000) = (1/1000) overhead

And RAM seems already at the limit....

I would think that having lots of different categories consumes more RAM... but honestly this is a guess, not data! And so, if one or 2 categories are quite larger than the others, I would probably try to single each of them, on the belief that RAM use will be more efficient and somehow processing too.

Are you importing pictures at the same time? that might be another source of processing time. You might consider importing pictures (and only them) at a later time: this will be an UPDATE of existing records, instead of part of the CREATION of a new record.

Hi fibo,

again thank you for the reply, I will test the bucket size increase and report back.

My process is this;

i upload my .csv import file to the var/import folder

Yes i am uploading pictures, they are in media/import folder

three image types,
 
folders as follows and 19,000 images in each

image200x200px - small_image135x135 - thumbnail -48x48
 
I don't know if the three different sizes are needed, but it is the procedure i have been using.

I will test an import with no images and measure performance

I am not sure of the update procedure for products.

what structure do i need the .csv do i leave just enter the data in the fields that need to be updated or must all data go in again + the new updated image fields?

Thanks for the help.

Assuming the products have been created, an update / complement csv file will need sku or, better, product_id, and yhe fields you want to update

this extension does everything you asked and then some --> http://www.magentocommerce.com/extension/1894

Where can I adjust this "bucket" size you mentioned?