06 Jan 2012 Smart Storage
Running scientific data management software directly on a storage appliance
The first picture shows a couple of Drobo storage arrays. The second image is a screen capture showing BioTeam running it’s own MiniLIMS product as an embedded application hosted directly inside the Drobo storage array. Cool huh?
The idea that an inexpensive and super easy-to-use storage appliance can easily be made to natively host scientific data management software is pretty fascinating. There are a ton of high-end storage solutions for large-scale genome platforms but you don’t hear a lot buzz about small, simple and clever things targeted at single researchers or single instruments.
This blog post is intended simply to highlight something cool that we were able to do while hacking and experimenting on our own time. The freedom to fool around with stuff like this is one of the reasons I love coming to work every day.
Just to be clear:
- This is not a product
- This is not officially sanctioned or supported by anyone
- It’s a neat proof of concept, nothing more.
Meh. Whats the big deal?
Storage systems have been adding internal horsepower rapidly for years now. This internal power is necessary to run additional services and features such as encryption, replication and data-deduplication etc. that the market has been demanding.
That’s old news. It’s also nothing new to learn that storage vendors have often been obtaining this performance by leveraging the commoditization wave seen with standard server technologies and chipsets. Sure they still differentiate themselves & juice their core storage engines with proprietary engineering, silicon and hardware but more and more of them are running core functions on standard tech that IT pros are well familiar with.
As the price/performance of mass-market server technology increases it becomes easier and easier to put “more power” into infrastructure items like network switches and storage arrays.
In addition to the increasing use of mass-market hardware inside these devices, more and more of them are running internal operating systems that are either 100% Linux or very “unix-like” inside. Anyone who has been root on an Isilon cluster or Panasas storage brick will have seen this first-hand.
Once you have hardware CPU cycles to spare and an internal OS that can pretty easily run common development tools, libraries and applications the next logical question becomes “what other software can I run directly inside this environment?”.
This, also, is not anything new. My first experience with this a few years back was seeing how Isilon embedded software from Aspera into their arrays so that their broadcast media clients could more easily sling HD video streams around the world.
Ok, so why the blog post?
Getting MiniLIMs to run as an application inside a Drobo storage array is cool and worthy of a minor blog writeup for the following reasons:
- Several vendors have “application aware” infrastructure & datacenter products but these are often targeted at big enterprises and for support/profit reasons they are often locked down to hosting only a certain set of software products. It’s hard to wedge your “own stuff” into these platforms
- DIY & rackmount Linux storage servers can easily handle this use case (for instance our 120TB Backblaze Pod) and there are tons of other options out there for people who want to build real servers that can both run applications and internally provide a ton of storage capacity. Companies like Silicon Mechanics have all sorts of suitable offerings. The problem? Most of these systems completely fail the “easy-to-use appliance” test. They are great systems but they need to be purchased, integrated and supported by IT professionals if they are going to succeed in a research or scientific environment.
- The notable thing in our mind is that we were able to place our scientific software inside a storage device that was designed from Day 1 to act as a simple appliance. No IT expert required. The Drobo units are notable for being dead easy to setup, manage and use. In addition, Drobo goes out of their way to encourage 3rd party developers to create “DroboApps” that run on their architecture.
It’s not inconceivable similar “smart storage” devices could replace traditional PC-based laboratory instrument control workstations.
As storage units get smarter and more capable the need for a dedicated Windows PC attached to an instrument or Genome Sequencer becomes less important.
We particularly like the idea of a small storage appliance that is capable of natively hosting scientific and data management software – there are a lot of instrument-attached use cases that come to mind. Something like this seems attractive for single-instrument genomics environments or labs where dedicated research IT staff may not be easily available.
I’ll end this post with a few more Drobo pictures just to really highlight their “small appliance” nature …