If you’re using the Datafeedr API, chances are you have ambitions of building a large store. Many of my clients come to me looking to build something like Shop Style which has somewhere in the region of 1,000,000+ products.
Adding that many products to WordPress using WooCommerce and the default settings and shared hosting will kill your site. Firstly – it will take Datafeedr a few weeks to add that many products to your store, but secondly most hosts just aren’t configured to run WooCommerce with that many products.
Below, I review what you need to check with your hosting provider to ensure your massive WooCommerce store is going to stay on its feet and serve your customers.
Quick summary
Use the Datafeedr API to load millions of products, use Digital Ocean for your hosting (get $10 credit with Digital Ocean with this link), get my WPI server stack and my WPI performance plugin and you can run stores with many millions of products with WooCommerce.
Storage Space
The product images are going to be what consume all the space on your server. If you want to store 1,000,000 products, or even just 100,000 products, you need to plan your storage requirements.
100,000 products = roughly 40GB of storage
This figure obviously varies among shopping sectors – the above is about right if you’re looking at clothing as the merchants tend to provide very high resolution images. On top of this, WordPress will make multiple copies of your images in order to provide them in multiple different formats – e.g. thumbnails.
Now – if you’re going for 1,000,000 products that means you need 400GB of storage which is a LOT but not infeasible in this day and age. Your best bet is to store your images on something like Amazon S3 where you can store images far more cheaply than using the disk space which comes with your server.
There is a great little plugin which will handle all of this for you – provided you install it prior to loading all of your products, it will copy all of your images over to Amazon S3 and delete the original images meaning you can pay Amazon for storing those images rather than your host. This will give you a faster site too since your host won’t have to deal with serving up images and instead can concentrate on serving up the pages for your users.
Note: The plugin above will migrate any newly created images over to Amazon – it won’t migrate any of your old images. If you’re using Datafeedr already, the way to get round this is to delete your product sets (temporarily, they move to trash), delete your orphaned images, then restore your product sets.
Speed of Disks
You really want Raid SSD, not HHD or SAN. This is because WordPress and Woocommerce use a lot of files and you need the fast seek speed provided by Raid SSD in order to get decent Datafeedr speed when you’ve got 100,000+ products.
Web Server
You want Nginx, not Apache. Apache sucks – as soon as you have a few users on your site it will slow to a crawl and die a death.
HHVM to make Datafeedr and Woocommerce really fast
If you have a lot of products in your affiliate store, you need HHVM. This is a PHP compiler built by Facebook. It makes WooCommerce operate a LOT faster, even if you’re allowing users to sort through ALL your 200,000 affiliated products. The only host that currently offers this is WP Engine. You can install HHVM yourself though. Follow my DIY installation guide to get HHVM set up.
Options available
You really have two options available – either use WP Engine or build your own HHVM/Mercury installation.
If you’re building your own Mercury installation, I highly recommend Digital Ocean but then it depends how tech savvy you are. If you’re not tech savvy – go with WP Engine but just be sure to install that Amazon S3 Plugin before you start loading products as storage space with WP engine is pricey.
Founder at Affiliate Web Designers
Dave has been programming since 6 years old and has been developing WordPress plugins, themes and websites since 2010. In the past he built the browse view technology for Skyscanner and now he helps clients with interesting website challenges.
He is always on at least one of his computers when he's awake, so get in touch and he'll get right back to you.
He is always on at least one of his computers when he's awake, so get in touch and he'll get right back to you.
Latest posts by Dave Hilditch (see all)
- My WordPress performance plugins and server stack have moved - July 31, 2016
- Price Comparison Pro 1.2 Released - July 5, 2016
- How to run backups on huge WordPress websites without your website being brought offline - February 4, 2016
February 5, 2015 @ 1:51 pm
Another great post addressing the issue of having more than 100K posts (ie. products) on your WordPress installation. Thanks for sharing, David!
Out of curiosity, do you run any large Datafeedr sites yourself? If so, where do you host them? What’s your set up?
February 8, 2015 @ 2:11 pm
Mostly running large datafeedr sites for clients – the best hosting I’ve found for them is on Digital Ocean using the Mercury installation.
e.g. an Israeli client has about 170,000 products in her Datafeedr store now – before I fixed things for her, the store was taking 190 seconds to load. After I’d implemented HHVM (part of the Mercury installation) that dropped to 1.5 seconds – that’s WITHOUT caching.
You can see the detailed DIY installation guide I made here: http://www.affiliatewebdesigners.com/2015/02/04/diy-mercury-installation-super-fast-wordpress-performance/
February 10, 2015 @ 7:34 pm
Wow! 1.5 seconds, nice job!
February 10, 2015 @ 7:49 pm
Yeah – not as fast as I’d like but it’s still on a VPS. There’s some underlying performance problem in WordPress core that needs to be fixed, but anyway – reducing the NON-CACHED performance to 1.5s means that page caching can then kick in and THEN you’re talking 100 – 200ms for page load.
Providing ajax baskets are used then entire page-caching for your main shop pages works great.
January 22, 2016 @ 12:40 pm
Could I ask which hosting company you use for this? I’m not convinced the ones I’m using know how to set this up. Thanks in advance.
January 22, 2016 @ 2:12 pm
Hi – I use Digital Ocean for all my servers and heartily recommend it. You can use this affiliate link for $10 off – http://www.digitalocean.com/?refcode=a03d7288da06 – their servers are great and affordable.
The best set up for affiliate stores is the Mercury stack – I provide a service to install that on Digital Ocean (or any Ubuntu 14.04 64 bit environment) – here’s the service: http://www.affiliatewebdesigners.com/shop/mercury-super-fast-wordpress-installation-service/
Or you can do it yourself if you follow the instructions here: http://www.affiliatewebdesigners.com/2015/02/04/diy-mercury-installation-super-fast-wordpress-performance/
March 2, 2016 @ 7:15 pm
Hi guys my name is kelvin might be thinking to use datafeedr api in order to run my affliate stores. I was wondering if i have a store that will have over 100,000 products what will be the best host for me to use & cheapest. I also want it to be very quick aswell so can you recommend me please.
July 5, 2016 @ 1:39 pm
Best to go with a host that uses SSD – I like Digital Ocean, but if you need support then go with WP Engine
November 23, 2016 @ 9:40 am
Hi Dave, great post. Could post here examples of your sites, to we see this performances, design etc. Thanks…
February 19, 2017 @ 6:04 pm
We’re building a suite of showcase sites at the moment – in the meantime you can check out one I hacked together over here: http://www.foundthru.com
January 8, 2017 @ 10:08 pm
Hi,
I have installed datafeedr on my WordPress website where I want to display a lot of affiliate products, for deals/coupons. How can I import the productfeed? Do I need to use woo commerce? I am kinda new to all of this.
February 19, 2017 @ 6:02 pm
With Datafeedr you get two options – the API works with WooCommerce while the Factory works on its own. I recommend the API because then you can import products using other tools as well as datafeedr, and you also can use all the woocommerce themes.