Monday, January 3, 2011

EMC, IBM, HP, NetApp and Sun - undeserving monopolists

Storage.

For those of us who use computers, it is the most boring and basic element of involvement.

Adding 200% of the current storage throughput available rarely gives us more than a 10% increase in real-life benefits thanks to applications being more interested in order of operations than scheduling, and operating systems being more interested in scheduling than order of operations.

With the rise and rise (and subsequent fall) of ZFS, the role of the FS to do the right thing in the best way possible without delegating responsibilities to logical volume managers, RAID controllers and/or remote storage targets has been realised (and then discarded).

Thanks to a combination of geeks focusing on issues that are "fun", application developers focusing on issues entirely within their constrained domain and Linux kernel developers ignorantly choosing their targets without any regard for the direct employers and (n + 1) indirect benefactors who make their employment possible (indirect FFS/extX FS inodes - you all know about such things), Linux is in a sad and embarrassing state in this area. DRBD is a great piece of software, but it may very well propagate file system problems as fast as your well-designed (and implemented) network will allow.

Don't take my word for it - there are plenty of people beyond my rambling self who agree.

So, as a direct consequence of this current shortcoming, I am announcing a project (name to be made public in the next week) that will provide architecture and OS independent CDP and near-CDP, with appropriate knobs to allow System Administrators to decide which trade-off they need to allow between performance and availability whilst allowing for implementation of local, near and remote DR in any conceivable combination.

In terms of implementation, the implementation couldn't care less which OS, filesystem or hardware you use. The implementation writes bytes to files as requested - exactly as write() and pwrite() do.

No comments: