s131

DIY Mercury installation for super fast WordPress performance

, , 28

I’ve just released a new service to set up, migrate and configure your WordPress site for you on Digital Ocean using the latest and greatest Mercury Install. BUT, maybe you want to set this up yourself? If so, read on for the full guide.

This includes:

  • Nginx – web server (far faster than Apache)
  • Memcached and APC – caching engine which works with W3 Total Cache
  • Varnish – HTTP accelerator
  • HHVM with fallback to PHP-FPM – compiled PHP for much faster pages
  • Percona DB – faster than MySQL (100% compatibility)

Installing just Nginx will make your site a helluva lot faster and able to support many many more simultaneous users, but add in all these other features and you’ll be the envy of all your WordPress friends.

If you want to see some charts and stuff, check out the High Performance WordPress Installation Service page.

DIY Mercury Installation Steps

You need to know SSH and some Linux commands in order to build this yourself. Here’s the steps:

  1. Get yourself a Digital Ocean droplet (or any Ubuntu 14.04+ VPS but Digital Ocean use Raid SSD so they’re a great choice) (this affiliate link gets you $10 off)
  2. If you’re migrating, point new.yourdomain.com at your Digital Ocean IP address. If this is for a new site, point yourdomain.com at your Digital Ocean IP address
  3. Follow the instructions using this Ansible Playlist to the letter
  4. Install cURL, mcrypt and set up your firewall – 
    sudo apt-get install php5-curl
sudo apt-get install php5-mcrypt
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 25/tcp
sudo ufw enable
  5. Set up PHP7 for your faster fallback plan: 
    sudo add-apt-repository ppa:ondrej/php
sudo apt-get update
sudo apt-get install -y tmux curl wget nginx php7.0-fpm php7.0-cli php7.0-curl php7.0-gd php7.0-intl 
sudo apt-get install php7.0-mysql php-memcached php7.0-mbstring php7.0-zip php7.0-xml
vi /etc/nginx/conf.d/upstream.conf
    (Change php socket to new php7 socket)

    upstream php {
 server unix:/var/run/php/php7.0-fpm.sock;
}
  6. Set up fail2ban to protect your server from attack (edit the jail.local file once you’ve created it, find the nginx section and change enabled to true): 
    sudo apt-get install fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

vi /etc/fail2ban/filter.d/wordpress.conf (add following lines)
[Definition]
failregex = ^<HOST>.*POST.*(wp-login\.php|xmlrpc\.php).* (403|499)
Then vi /etc/fail2ban/jail.conf
Change nginx enabled=true (instead of false)
Change bantime at top to 86400 (1 day instead of 10 minutes)
Add the following after nginx section:
[wordpress]
enabled = true
port = http,https
filter = wordpress
logpath = /var/log/nginx/*access.log
maxretry = 3
  7. Set up sendmail (not needed if you use an external service like Mandrill or Mailgun – note: more work needed to set up SPF records and make email get past Junk folders – use external service if in doubt) 
    sudo apt-get install sendmail
sudo sendmailconfig (choose Y and Y for answers to the questions)
  8. Set up unattended-upgrades (add “kernel”; to the blacklist section because if you are on Digital Ocean, you should update the kernel through the console to ensure snapshots and other things continue to work): 
    sudo apt-get install unattended-upgrades
vi /etc/apt/apt.conf.d/50unattended-upgrades
  9. Run 
    update-rc.d xxx disable (where xxx is the service you wish to switch off)

    for any services you do not wish to run. Check services with service –status-all. Note: If you have 1GB RAM or less on your server, switch off Varnish and HHVM and use PHP7 – you will need to change your /etc/nginx/sites-available/domain.com file – change three things: 1) Change port to 80 instead of 8080, 2) Change primary method from hhvm to php, 3) Remove the fallback line in the primary method

  10. Set up a proper cron job – locate your wp-config.php file (/var/www/html/yourdomain/wp-config.php) and add: 
    define('DISABLE_WP_CRON', 'true');
Then run crontab -e and add:
*/1 * * * * wget -q -O – http://YOURWEBSITE.COM/wp-cron.php?doing_wp_cron
  11. Change your timezone: 
    sudo dpkg-reconfigure tzdata
  12. Optionally configure longer timeouts and debug – useful if you’re going to be running large imports and such stuff
    Edit wp-config.php:

    define ('WP_DEBUG', true);
    define('WP_DEBUG_DISPLAY', false);
    define('WP_DEBUG_LOG', true);
    Edit /etc/nginx/sites-available/domain.com:
    add fastcgi_read_timeout 120; to your location block
    Edit /etc/php/7.0/fpm/php.ini:
    Change max_execution_time to 120
Change upload_max_filesize to something better than 2M
  13. Increase the number of child processes that get spawned:
    Edit /etc/php/7.0/fpm/pool.d/www.conf and change:

    pm.max_children 25
    pm.start_servers 5
    pm.max_spare_servers 5

    (you can play with above settings depending on the RAM on your droplet)

  14. Increase the number of connections available to nginx – edit /etc/nginx/nginx.conf – something like this at the beginning: 
    events {
     worker_connections 20000;
     multi_accept on;
     use epoll;
    }
    worker_rlimit_nofile 42000;
  15. Configure MySQL – vi /etc/mysql/my.cnf [mysqld section] (depends on memory available, engine used, and what you’re doing with wordpress) 
    key_buffer = 512M
max_allowed_packet = 512M
thread_stack = 192K
thread_cache_size = 8
query_cache_limit = 512M
query_cache_size = 4192M (or 0 can improve performance, it depends)
innodb_buffer_pool_size = 20000M (according to your memory - as a guide, use 60% of your memory here)
innodb_buffer_pool_instances = 16 (according to how many cores you have)
innodb_io_capacity = 5000 (because Digital Ocean use fast SSDs so we don't want InnoDB to be limited)

Restart your key services (nginx, varnish, hhvm, php7.0-fpm) or reboot your server with shutdown -r now. Your new server and wordpress installation is now ready, fast and secure.

Migrating your existing site

  1. Log onto your original server
  2. cd to your wp-content folder
  3. Create a backup of the mysql database using: mysqldump -u root -p dbname > sqldump.sql
    1. If you don’t have SSH access, create a template file and add the following code: <?php shell_exec(‘mysqldump -u root -p dbname > sqldump.sql’); ?> then create a page with that template and view the page – you should now have a backup file – this is useful to get round stupid hosts like Hostgator or Godaddy if they’ve disabled SSH and backups
  4. Log onto your new server
  5. cd to your wp-content folder
  6. Connect with FTP to your old server: sftp root@ipaddress
  7. Get everything using: get -r *
  8. Exit FTP
  9. Restore your database using:
    1. mysql -u root -p
    2. create database dbname;
    3. use dbname;
    4. source sqldump.sql; — replace sqldump.sql with the full path to your sqldump.sql file (e.g. /var/www/html/websitename/wp-content/sqldump.sql)
  10. Exit mysql and delete the sqldump.sql file
  11. You might need to copy across the contents of your wp-config.php file – have a look at the differences for both using FTP.
  12. Update /etc/nginx/sites-available/domain.com to point to the folder you downloaded from your old server
  13. Once you’re happy, switch your DNS settings to point your domain name at your new server

Set up W3 Total Cache

  1. Log onto your new website
  2. Install W3 Total Cache
  3. Set up page caching, minification, database caching all using Opcode: APC caching. There is a complete guide to setting up W3 Caching here BUT use Disk:Enhanced only for page caching – use OpCode:APC Caching for everything else. If you’re using Datafeedr (or for maximum compatibility with other plugins), don’t add object caching.
  4. Optionally set up Cloudflare or similar CDN – if you do, also install the Cloudflare plugin as it fixes some peculiarities with IP address origination
  5. Add the following to your wp-config.php file:

Configure email

Edit /etc/hosts – find the 127.0.0.1 line and add your fully qualified domain name (hostname.yourdomain.com):


127.0.0.1 hostname.yourdomain.com hostname

Then run the following:


apt-get install exim4-daemon-light mailutils

dpkg-reconfigure exim4-config

From the screens that follow, choose:

  • Internet site
  • Enter your fully qualified domain name
  • Add your hostname and localhost to the list of servers that can send (as well as the fully qualified domain name) separated by semi-colons
  • Leave ‘Domains to relay for’ and ‘Machines to relay for both blank
  • Say ‘No’ to ‘Keep number of DNS-queries minimal (Dial-on-Demand)’?
  • Leave other options at their defaults

Now your server can send mail which doesn’t bounce from WordPress.

That’s it – you now have a superfast WordPress installation – the fastest it POSSIBLY can CURRENTLY be for your server configuration.

The above runs on $5pcm Digital Ocean upwards, although $20pcm+ is recommended otherwise you’ll need to disable some services – e.g. only use one engine (HHVM or PHP7), disable varnish and disable memcached. You still get great performance and better than anything else you’ll find as this is tuned for WordPress.

If you don’t want to follow the steps above, don’t have time, or are confused, I’ve listed a service where you can pay us to set this up for you. Get that fast wordpress installation service here.

Adding a new website

The best way to add a new website is to copy the file in /etc/nginx/sites-available and configure the new file as per your new site.

Chat to me
Latest posts by Dave Hilditch (see all)

Dave Hilditch

Dave has been programming since 6 years old and has been developing Wordpress plugins, themes and websites since 2010. In the past he built the browse view technology for Skyscanner and now he helps clients with interesting website challenges. He is always on at least one of his computers when he's awake, so get in touch and he'll get right back to you.

→ Dave Hilditch

Related Posts

28 Comments

  1. Affiliater
    February 22, 2015 @ 10:50 pm

    Thanks Dave for this exceptional article! It works like a charm and i’m really excited to get my datafeedr API site up and running now on Digital Ocean.

  2. Simon Haynes
    March 26, 2015 @ 3:25 pm

    This setup is fantastic, our sites are running much much faster! Thanks!

    P.S. have you had any issues with this setup and the Datafeedr Bulk import tool getting stuck on the same image/product?

    • Dave Hilditch
      March 26, 2015 @ 3:32 pm

      One thing to watch out for with Datafeedr is to ensure you don’t enable ‘Object caching’ in W3 Total Cache as it’s not compatible.

      I’ve not experienced Datafeedr Bulk Import tool getting stuck on same image/product – make sure you have a proper cron job set up so you’re not relying on visitor traffic in order for your batch jobs to run – I made a guide for that here: http://www.affiliatewebdesigners.com/2014/11/22/set-real-cron-job-datafeedr-woocommerce-product-sets/

    • Eric
      May 4, 2015 @ 6:55 pm

      Hi Simon

      This issue should be fixed in the newest version of the Datafeedr Product Sets plugin. It was an issue related to caching plugins and the Object cache.

      Eric

      • Dave Hilditch
        May 4, 2015 @ 9:27 pm

        That’s great news. I’ve got a client I’m setting up the mercury service for, so I’ll include the object caching and we can see what impact it makes.

  3. Simon Haynes
    March 27, 2015 @ 2:15 pm

    We have this running on an 8gb 8 core server, what in your professional opinion would be the optimal hhvm settings for memory etc?

    • Dave Hilditch
      May 4, 2015 @ 9:30 pm

      Wow – sorry I missed this comment. I’d need to measure the impact – it can depend on the size of your database, or how long your pages are taking to process. I’ve got plenty sites with 256MB script setting in HHVM that are running very well. You can track your uptime using http://www.uptimerobot.com for free as well, which is important if you’re scaling.

  4. judo
    February 10, 2016 @ 8:16 am

    i tried to install this diy mercury stack to digitalocean droplet with your kind explanation. and i think it was done succesfully .

    but i couldn’t upload my wordpress theme . i did almost everything in the google so that solve this issues but it’s still not work.

    what can i do for it?? can you give me some advice?

    • Dave Hilditch
      February 10, 2016 @ 9:56 am

      Hi – is there any error message specifically? If you are saying you can’t upload your wordpress theme, I presume that means you got WordPress installed. Probably it’s your /etc/php/7.0/fpm/php.ini file – if you edit this you will find a line like this:

      upload_max_filesize = 2M

      Change the 2M to 24M and you should be able to upload your theme. That’s just a guess at what’s wrong as you didn’t give me very much to go on!

      Or it could be your /etc/hhvm/php.ini file – same line (you will need to ADD it to the hhvm/php.ini file as it doesn’t exist in the first place). Depends what you have set up in your nginx /etc/nginx/sites-available/yourdomain.com file to process your PHP (either PHP 7 or HHVM)

      • judo
        February 10, 2016 @ 2:42 pm

        thanks for your reply. the exist message from wordpress uploader was ‘are you really do this?’

        i did edit php.ini php7.0 folder. and it didn’t works. i will try same things in hhvm. thanks a lot!

      • judo
        February 10, 2016 @ 4:54 pm

        Now, another error code has been generated during uploading theme.

        Error 503 Service Unavailable

        Service Unavailable

        Guru Meditation:

        XID: 1771580132

        Varnish cache server

        should i off varnish??

        • Dave Hilditch
          February 10, 2016 @ 7:07 pm

          If you have varnish enabled, make sure you access your back end with :8080 port which bypasses varnish.

          e.g. http://www.yourdomain.com:8080/wp-admin/

          Alternatively, disable varnish using the disable command above and:

          service varnish stop

          And then edit the port number in /etc/nginx/sites-available/yourdomain.com to 80 instead of 8080.

  5. Marc
    February 13, 2016 @ 9:06 am

    Awesomesauce! Followed your instructions, step-by-step.

    I got a bit confused as to how to change the configuration to disable Varnish and HHVM… but I think it’s working. *optimistic*

    • Dave Hilditch
      February 14, 2016 @ 4:44 pm

      Good – glad to hear it – let me know how it goes. I’ll keep working on improving this article to make it easier to follow for future readers.

  6. judo
    February 14, 2016 @ 3:09 pm

    hhmm i was destory my droplet on digital ocean. then when i tried to reinstall mercury stack to droplet , during the process for playbook.yml

    the message like this had appeared and failed.

    • Dave Hilditch
      February 14, 2016 @ 4:37 pm

      Judo – what message are you seeing when running the playbook? If you’ve got error messages, the likeliest scenario is you’re trying to install to a different operating system other than Ubuntu 14.04 64 bit – e.g. maybe you went for 32 bit by mistake?

  7. Cala
    February 14, 2016 @ 4:52 pm

    Hi, Dave.

    Thanks for the guide. It’s truly awesome!

    This mercury install uses and outdated version of nginx and varnish (v3 instead of v4).

    How can I use this mercury installation with the latest version of nginx and varnish (v4 sintax is not compatible with the v3 syntax).

    Thanks for your time!

    • Dave Hilditch
      February 14, 2016 @ 7:17 pm

      Hi – one way to get the latest version of nginx and varnish is to run:

      apt-get update
      apt-get upgrade

      Soon I’ll be making my own ansible script – maybe next week, maybe the week after – the main thing I’m aiming for is obviously the latest versions of nginx and varnish but most importantly I want nginx to be built with the ngx_http_realip_module built into it. You have to build from source to get this. It’s required if you’re using Cloudflare and Fail2ban together so that Fail2ban doesn’t end up banning the Cloudflare IP addresses.

      • Cala
        February 15, 2016 @ 3:29 pm

        That’s great news!

        I was able, in a clean Ubuntu installation, to first install nginx (latest version) and only then run the ansible script for the mercury environment. This seems to run just fine.

        Unfortunately this doesn’t work for varnish because when I run the mercury installation it fails because the syntax of the config file of varnish is incompatible with version 4.

        Regarding SSL (https) I was able to configure nginx as a proxy for 443 port so I can have both nginx and varnish running on https since varnish doesn’t support SSL. It works fine! But I have an issue… When trying to log in in wp-admin the port defaults to 8080 (I’m using this is to bypass varnish in the backend) but because of that SSL stops working (it works fine in the frontend). I get this error messages in Chrome: SSL connection error and ERR_SSL_PROTOCOL_ERROR. Still trying to figure this out :/

  8. Cala
    February 15, 2016 @ 3:33 pm

    Also, when you make your ansible script, please add the pagespeed mod and the newest version of openssl 1.0.2 so we can have ALPN when using http/2. That would be a great addition 🙂

    Thanks once again for all the contributions you’ve made. The mercury environment is awesome!

  9. Ferri
    March 12, 2016 @ 5:38 pm

    hi, whats the meaning of this?

    worker_rlimit_nofile 42000;

    and how to adjust pm.max_request correctly?

    thanks

    • Dave Hilditch
      April 5, 2016 @ 11:19 am

      Hi – the worker_rlimit_nofile controls how many files can be opened at once by worker processes for nginx. This is important if you’re using a disk/file-based caching system like that offered by W3 Total Cache.

  10. Mohammad Farouk
    April 14, 2016 @ 12:00 am

    Hello,

    Great tutorial, But I need to upgrade the current Varnish 3.0 installed through guide here to Varnish 4, can you please help with that ?

  11. matthew
    June 2, 2016 @ 2:26 am

    I think you missed what should be add in the wp-config in

    Set up W3 Total Cache

    5: Add the following to your wp-config.php file:

    • Dave Hilditch
      July 5, 2016 @ 1:33 pm

      Yes – you’re right – to complete your W3 Total Cache config if you visit your site /wp-admin/admin.php?page=w3tc_install it will tell you the extra lines to enter to your nginx config file

  12. Peter Ode
    June 30, 2016 @ 8:46 pm

    Hi Dave,
    I’ve been around Windows Servers since NT and 2000 versions, hosted on my own hardware. Now I’m learning Linux and love being able to spin up a $5 or $10 VPS at Digital Ocean in a few minutes.

    I’m mostly hosting WordPress sites and find them on the slow side. I will also be installing WooCommerce on WP. Your Mercury stack seems the way to go. I have a few questions that might also benefit your other readers.

    Q1: You discussed a newer Ansible Playbook that includes the newer NGINX and Varnish. If now available, can you please direct me? Otherwise, do you have any suggestions about how to update your tutorial to get the latest please.

    Q2: You state only Ununtu 14.04 should be used. It’s now a year later so I’m wondering if Ubuntu 16.04 can work?

    Q3: I would like to use the AJENTI.org control panel. Where in your tutorial should I install Ajenti?

    Q4: With respect to WordPress and all the plugins, is PHP 7, for the fallback, better than PHP 5.6? I saw your post about fixes for major plugins for PHP7, but that leaves other plugins I might use with problems (potentially).

    Thank you.

    • Dave Hilditch
      July 4, 2016 @ 4:11 pm

      1. I don’t have it to hand right now and google is failing me – you can run this tutorial and then upgrade varnish and/or rebuild nginx if you wish

      2. I haven’t yet tested it with Ubuntu 16.04 – I’d love it if someone would try and feedback?

      3. I don’t have Ajenti installation in the tutorial yet. It’s something I’m offering for people who purchase the installation service but I can tell you it’s not that easy or straightforward to install. I’ll try and pull a guide together, but I may yet back off from Ajenti if I can’t get it to do everything I want.

      4. It’s your choice – you can have PHP 7 + PHP 5.6 fallback if you wish, or HHVM + PHP 5.6 fallback. This ansible script installs PHP 5.6, so you will have that available as an option if you wish.

Leave a Reply