Deploys at WordPress.com VIP

About me: Mohammad (Mo) Jangda

[email protected] | [email protected] | @mjangda

• Code Wrangler at Automattic
• Ice Cream Fan

What is WordPress.com?

Hosted platform for running blogs, sites, etc. using WordPress.

Big multisite network.

What is VIP?

WordPress hosting and services for big companies (CNN, TIME, ESPN, CBS, TechCrunch, Williams, etc.)

A VIP site on WordPress.com is the same as a free site, except with some custom code a bit of magic sauce.

The VIP team works with external developers providing code review, developer support, etc.

Fancy Numbers

54 million sites
62 million users
30 million posts (per month)
13 billion pageviews / 400 uniques (per month)
500 million MySQL tables
2500 servers
3+ DCs

How Many Deploys?

Yesterday: 259

VIP: 165
WP.com: 75
Other: 19

How is code deployed on WordPress.com?

Production servers run trunk

– Commit changes to trunk
– Run deploy script
– Deploy script syncs svn mirrors across DCSs and runs `svn up` on all servers

What about VIPs?

Similar to WordPress.com except:

– Code comes from external developers
– Deploy done internally
– We only push the particular folder (code changes are limited to the “theme”)

VIP Numbers

5 Software Engineers; 1 Happiness Engineer; other biz people
7m lines of code (on top of WP.com codebase)
1000s of sites
~100 active developers
Over 140K commits (currently r140321)
70K deploys all time
Avg deploy time (commit => review => deploy): 130 minutes

Challenges

– External developers writing PHP, HTML, and JS so anything is fair game
– Issues on one site can spill over to another (also helps with scaling)

Code Review is Essential

Make sure they’re not doing not-so-good things

Guidelines: http://vip.wordpress.com/documentation/code-review-what-we-look-for/

Challenge: Code Review is hard and time-consuming! Blocks business needs!

Deploy Page

– custom built tool to optimize review process
– Used to track pending commits, review changes, and deploy
– Can send feedback on issues, alert rest of team when issues spotted
– Handy revert commands!
– “real-time”

Pre-Deploy Tests

Run when deploy button is pressed.

Loads up the site against a sandbox server in production with live database (crazy!) and latest code.

Static Analysis

Using custom scanning tools + PHP Code Sniffer to catch things like restricted functions, bad coding patterns/practices, etc.

Coming soon: post-commit tests

Using Jenkins to run tests immediately after commit, instead of on-demand using Sandbox server in production.

Other Tools

– Reference Page with handy revert commands
– IRC Channel piping PHP and MySQL errors from 2 servers
– IRC alerts for internal commit and deploy notifications
– Email and webhooks for external commit and deploy notifications

Future: Scaling

– More developers and automation
– Externally initiated deploys and post-deploy reviews

Thanks!

[email protected] | [email protected] | @mjangda

Hiring, Hiring, Hiring!

• VIP Wranglers, VIP Wranglers, VIP Wranglers!
• Code Wranglers, Code Wranglers, Code Wranglers!
• Designers, Designers, Designers!
• Happiness Engineers, Happiness Engineers, Happiness Engineers!