“Own the Stack”
Office #2 - Co-Working at WorkSpace (Photo Credit: Jean Cloutier / IN2CC)
Now that I’ve set out on my own, my plan has been to set aside a couple of months to solely work on “my project”, without the distraction of getting involved in any freelance client work. I think that it has been a great idea to set aside time to do this, as it has really allowed me to really catch up on what’s happening going on out there. I only wish that I was able to set aside even more time. It’s been about a month so far, so it’s time for an update.
So far, most of what I have been doing in the past month falls under the category of “triangulation”. Mostly, I have been looking at what is happening in various open source areas, and I have been digging deep into the projects which are of particular interest to me. I have some very ambitious goals, so I have to be strategic about what I take on. I’m very excited by what I have found so far.
I’ve got one more free month. Then, starting in October, the plan is to start doing freelance contract work for various clients in the Vancouver area. I’m very encouraged by the amount of interest I’ve received so far. In September, I am going to concentrate on improving my infrastructure, so I can quickly “spin up” a development environment to work on any component of the software stack that I support. As Brian Shire said at his OpenWeb Vancouver talk, it’s important to “own the stack”.
The “stack” that I am interesting in “owning” is quite a large one, encompassing all of the software I use on a daily basis. I run a server with Xen and Linux, plus I have some additional inexpensive VPSs, and I want to incorporate various “cloud” services such as Amazon’s EC2/S3 and Google App Engine (but without the vendor lock-in). On top of that, I want to target the client side in a cross-platform manner. That means utilitizing “Open Web” technologies and working with Web Standards and Javascript. And beyond that, there is still the need to build native client-side applications and virtual machine environments for the mainstream platforms - Win32, OS X, Linux. Plus there’s all manner of interesting embedded platforms out there (I have an iPhone now).
There’s a lot of work there. I’m ambitious, but I’m also a pragmatist. There are a few tools that are missing that would really make it easier to digest such a huge a amount of software. My first “project” is going to revolve around plugging some of those gaps. I’ll try to sketch out the first big problem I’m going to try to tackle.
Most software these days is characterized by the following:
- Very deep stacks, from the UI, through the network, all the way down to the OS and underlying CPU microarchitecture
- Lots of virtualization, so parts of the stack can be interchanged
- Lots of modularity, with lots of plugins or drivers
- A wide selection of very good languages and APIs to choose from
- Less optimization, because the chips just keep getting faster
There is a whole lot of software behind a typical application these days. Even seemingly trivial applications may invoke the work of literally thousands of previous contributors at various levels of the stack. The complexity is incredible.
The most interesting software that I am finding lately is all at the “edges” of various free software / open source projects - typically, a developer builds a plugin for use in a prototype of a system, and just tosses it out there. Unfortunately, most plugin authors do not have the time or resources to build a full project around their creation, so usually the code just sits out there for a long time until it gains some adoption. More often than not, the code does not get immediate adoption (even though it might be really good) — meanwhile, the underlying APIs and interfaces it is built upon continue to evolve and change. There is an awful lot of contributed code out there that is really cool, but has been afflicted with “bit rot“, so it takes extra effort to get it working again with the latest stuff.
There’s a simple solution to the “bit rot” problem — build unit tests, and do continuous integration.
The big problem is that most continuous integration platforms are not at all geared towards the needs of individual plugin authors (or users). Generally speaking, all of the available continuous integration systems involve a huge amount of setup and are oriented towards use with a centralized code repository. With the advent of distributed version control systems such as git, mercurial, bzr, darcs and monotone — more and more development is moving to the “edge”. It would be really nice to have a “distributed” continuous integration system that would have less of an “impedance mismatch” with the new world of distributed version control systems.
I don’t really have a name for this new project yet. Although, I imagine a good slogan might be “Continuous Integration for the Long Tail”.
For the next post, I’ll talk a bit more about some of the really cool technology I’m going to attempt to do the front-end UI with.
2 comments August 30th, 2008
