Deploying Applications with zc.buildout

Background

• For more than 10 years, building Python-based content-management systems and retail web sites.
• Lots of moving parts
• Automation of deployment, originally to customers environments and now to our hosting environment.
• make-based deployments were painful.
• Hostile to operations

make pain

• make uses shell scripts as it's scripting language.
  • Maintenance a disaster.
• Wanted to express rules in Python
• At high level, Wanted a configuration model, rather than a programming model.

setuptools and easy_install

• Wanted to manage Python modules more effectively

• Component-based Zope 3 still became a monolith. Gaaaa!

• Easy_install (still) was geared to installing into shared Python

  (Virtualenv helps a lot, but...)

• Needed more predictable outcome

  Building 2 checkouts of the same project should produce the same result, regardless of history.

zc.buildout

• High-level configuration using an ini-style syntax (ConfigParser)
• Recipes written in Python
• Automated assembly of Python project distributions with predictable outcomes.
• Automate un-installing!!!

Be kind to operations

• standard system packaging
• standard file locations
• standard process management
• monitoring support

Software installation

• System packages (RPM in our case)
• Install separate from configuration
• Installed software includes the tools needed for configuration.

Example CMS Application

• development buildout
• source release
• RPM generation
• deployment configuration

development buildout.cfg

Software deployment

• Source release
• RPM from source release
• Install RPM
• Example: smdappdb

Generating a source release

• source configuration

• buildout-source-release:

  bin/buildout-source-release \
     -n cmsappdb-0.1.0 \
     svn+ssh://svn.zope.org/repos/main/Sandbox/J1m/pycon2011/dev \
     dbsource.cfg
  

• produces cmsappdb-0.1.0.tgz

dbsource.cfg

Generating an RPM

• Copy source release tar archive to RPM source directory.

• spec file

• Run rpmbuild:

  rpmbuild -ba cmsappdb.spec
  

• Produces source and binary RPM

• Install:

  sudo rpm -i cmsappdb-0.1.0-0.x86_64.rpm
  

cmsappdb.spec

Configuration

• After software is installed in production, configure "deployments".

• Create buildout configurations that install configuration files, startup scripts, cron jobs, and so on to implement an application.

• system buildouts (zc.sbo):

  /opt/cmsappdb/bin/sbo cmsappdb ample
  

/etc/cmsappdb/ample.cfg

Installed paths

/etc/ample
/etc/ample/gc-config
/etc/ample/index-zdaemon.conf
/etc/ample/index-zeo.conf
/etc/ample/main-zdaemon.conf
/etc/ample/main-zeo.conf
/etc/ample/pack.sh
/var/log/ample
/var/run/ample
/etc/init.d/ample-main
/etc/init.d/ample-index
/etc/init.d/ample
/etc/cron.d/ample-pack

Meta recipe

• New customer, EX. Want a simplified configuration:

  [buildout]
  parts = ex
  
  [ex]
  recipe = cmsappdbmetarecipe
  main-port = 8200
  pack-time = 1 3 * * 6
  

• Meta recipe written in Python, just like other recipes

• Generate configuration by adding parts

cmsappdbmetarecipe.py

Managing whole applications

• What we have is good for deploying individual services
• Not so good for whole applications
• People have to make all of the connections
• Configuration tools like Puppet and BCFG can help, but still ultimately a matter of managing files.

Future work: makemea

• Deploy whole applications
• Application-defined models
• Model generate configurations
  • Applied to networks
  • Stored in database
  • Transactional through actions and undo actions
• Push model
  • SSH
  • Controller application run by user rather than web application

Questions?

These slides:

On PyCon web site and at http://www.riversnake.com/pycon2011/

Longer version:

http://www.riversnake.com/pycon2011/full