First things first …

Everything I’m writing about here was done by BioTeam’s own Adam Kraut – our infrastructure orchestration, cloud, HPC and ruby ninja. As I slowly degrade into a pile of useless management sludge, it’s basically Adam who is shouldering the load of doing cool and clever stuff for which I am ever grateful.

Integrating Chef with MIT’s StarCluster

Huh?

In this post we plan to demo how we use MIT StarCluster to deploy self-organizing GridEngine clusters in the cloud in a way that also lets us use a different system (“Chef“) to customize, control and install additional software.

OpsCode Chef

Chef is a really cool system for doing “systems orchestration” that BioTeam has been evangelizing & using for a while now. We use it to bootstrap cloud server nodes and turn them into whatever we need at the time – anything from installing a simple scientific software package up through assembling and controlling a complex scientific workflow setup. It has a nice Web interface as well as a fantastic command line client (“knife”) which as nerds, we wholeheartedly approve of.

Tools like Chef remove the need to hand-craft cloud server nodes stuffed with unique software, configurations and settings. If we ignore ALL the other features and advantages of Chef we still come out ahead with this feature alone.

Related blog post: “Coffeshop Cloud Orchestration” — another video showing Chef in action bootstrapping an Amazon cloud node.

MIT StarCluster

StarCluster does what BioTeam used to do, only better, slicker and with more committed developer resources, heh. It’s an open source stack written in Python that builds fully functional and ready-to-use Grid Engine clusters inside the Amazon Web Services Elastic Compute Cloud. StarCluster goes way beyond just installing Grid Engine – it comes preloaded with working MPI, scientific computing applications and NFS filesharing between cloud nodes. It also has nice features for automatically creating, attaching and sharing EBS volumes that might contain your applications, home directories and input/output data. It’s a great system and getting better all the time.

The StarCluster team took an idea that we had been fooling around with since 2008 and actually turned it into something that lots of people can use, customize and extend. In my 2008 and earlier experiments & talks about using SGE on the cloud I never really got beyond developing simple methods and scripts that suited only my needs.

Why combine Chef & StarCluster?

It’s really the best of both worlds for us. We love StarCluster because they’ve done all the hard work in sorting out how to launch, control and deal with self-organizing Grid Engine clusters on the cloud.

At the same time, however, we are working on so many projects and doing so many things that we really can’t bake ‘our stuff‘ into static EC2 server AMI images or even EBS volumes. Too much of an operational and maintenance headache.

Another example — Customer A may only want to deal with CentOS Linux while Customer B might be a Debian or Ubuntu fan. Chef makes us agnostic when it comes to most OS platforms — we simply don’t care and don’t have to manage lots of different OS and distro variants.

We prefer to write reusable Chef “recipes” and “cookbooks” for what we need and store those cookbook files under a source code management system like git or SVN so we have change-control and an audit history.

Our “infrastructure” is managed and treated like source code. It’s a thing of beauty.

Whatever our needs are — from installing the Illumina analysis pipeline to uploading a selective set of SSH keys for a client to login to a certain node — Chef can handle it with ease.

Screencast Video: Bootstrapping StarCluster with Chef

What does the video actually show?

• Adam Kraut using Starcluster on his laptop to fire up a single-node Grid Engine system on EC2
• StarCluster does all the hard work of standing up and configuring the cluster
• Once the Grid Engine system is up, Adam uses the Chef “knife” command to remotely install the Chef-client software and appropriate configuration files and encryption certificates.
• The end result is a Grid Engine cluster assembled by StarCluster while still being fully-managed Chef clients that we can make dance to our tune

A couple of things about this recording:

• It’s hard to get the terminal, recording and streaming settings correct for a terminal window that needs a wide area in which to output text. For the embedded youtube.com video below we recommend adjusting the settings to display at 720p “HD” resolution. Running it full-screen or expanded will also make the text more readable.
• If the text is still unreadable, you can directly download the original 30MB quicktime .MOV file at this link

Below the video…

I’ll upload some additional static screenshots showing “knife” etc. in action…

Additional Screenshots

Partial view of the Chef web interface

Some simple Chef command-line interaction using the “knife” utility

Knife & Chef understand “the cloud”, in this case we are asking the Chef server to list all EC2 nodes under management

Running the chef-client software on a cloud node. This node has already processed it’s “run list” so there are no more activities it needs to do

StarCluster command line client example

This is a quick blog post to highlight the types of projects I love working on. In this case we are standing up a dedicated Chef Server on the cloud and using it to orchestrate cloud systems. Sounds boring, right?

The cool bit is that we are not doing any of the time-wasting actions of customizing cloud server images to bake in software that we want or load it with pre-shared management software or credentials. Creating a wild pile of custom private AMIs that we have to maintain, manage and update is not something we want to take on if we can avoid it…

Nope, we are starting with nothing but public Amazon EC2 server images. Without getting out of my chair at Starbucks I can:

• Launch any public CentOS, Ubuntu or Amazon Linux AMI
• Remotely install the Chef-Client code (including any missing dependencies!)
• Automatically register and authenticate the new cloud node as a Chef-managed system under our direct control
• Use Chef Server to orchestrate the new node into doing whatever we want or need at the time

Boring: Launching cloud nodes from the command line

It’s pretty easy these days to start cloud nodes from the command line, especially if one already has hand-crafted server images sitting in the cloud waiting to be launched.

Yawn…

Starting & stopping servers is not all that interesting.

Even worse is wasting time hand-crafting custom servers stuffed with software or configured to perform unique roles or activities. Maintaining and managing a wild pile of AMIs is an operational headache that does not scale well.

Badass: Hands-free, barebones cloud orchestration

Please forgive the hacked up screencast, I did not have Camtasia handy today so I used a different, less familiar recording package. The video below is available in 720p “HD” but you might have to explicitly select that playback option if it defaults to 360p.

The video below shows:

• Using the Chef command line tool “knife” on my little macbook to interact with our Chef Server
• Using “knife” to launch a generic public Amazon Linux system on a m1.small EC2 server instance
• Remote installation of the Chef-client software stack
• Remote installation of the chef configuration files and certificates that allow it to securely connect to our Chef Server
• Automatic registration with the Chef Server as a node ready and waiting to be remotely controlled, orchestrated or provisioned

Comments welcome, I’m still trying to figure out if Youtube will work for Screencasts or if we are better off sticking with www.screencast.com. The video below is available in 760p HD format so adjust the player settings the text looks unclear or blurry.

2011 BioIT World Slides

As promised, here are the talk slides from the 2011 Molecular Medicine TriConference where we offered a short course titled Applying Next-Generation Sequencing Technologies: Introduction to New Technologies and Application in Research. To download the slides on “Agile Infrastructure in the Sequencing Era” click on the image above or use this link Kraut-TriConference-2011.pdf.

Practical Amazon Virtual Private Cloud (VPC)

A few things aligned last week – namely the fact that I was playing around with a newly acquired Juniper SSG-5 security gateway device at the same time I was testing both Amazon VPC and a new upgrade to some Camtasia Studio software.

The end result is a set of screencasts talking about Amazon VPC and how it is now in reach of smaller organizations now that Amazon is officially supporting smaller and more affordable VPN endpoint hardware.

• Screencast #1 – This is a narrated powerpoint presentation talking about what VPC does and what it should (and should not!) be used for
• Screencast #2 – This is also a narrated presentation where we provide more information about the actual device and how we plan to configure it for our VPC experiment
• Screencast #3 – A proper screencast showing the “live” setup and deployment of a VPC connection between my home lab and Amazon Web Services

The presentations were produced for viewing at 800×600 resolution making them too large to embed within this blog post. Click on the pictures below to be taken to the screencast.com hosting site. If you ‘hover’ your mouse over the flash movie, a table of contents will appear.

Practical Amazon VPC – Part 1 of 3 – Intro

Feedback and comments are welcome, we are still kicking the tires on various tools and technologies for online training and technical writing. Let us know what you think.  Does the 800×600 resolution mean that the file is too large to easily stream or watch?

As promised, here are the talk slides from the 2010 HPC Biomedical Leadership Summit where I was part of a morning talk session focusing on storage, large data and peta-scale data management issues. There were a number of interesting speakers but we were all limited to about 10 minutes duration so this slide deck is lighter than my usual stuff. To download my slides on “Grunt’s Eye View: Storage 2010″ please click on the image above or use this link: 2010-HarvardHPC_cdag_v2.pdf

As promised, here are the talk slides from my 2010 talk at the Beyond the Genome Conference. Click on the image above or follow this link: 2010-BeyondTheGenome_cdag.pdf

Apologies for the delay in posting this PDF. For some reason I can’t shrink this down to a reasonable size, possibly because of the amount of pictures included in the original presentation. The PDF is about 19MB (sorry!) and I’ll try to shrink it later if I can. Click on the image above or follow this link to download it: 2010-BioITWorld-Europe_v6.pdf