Sunday, May 31, 2009

AVS is a really bad idea

For those not familiar with AVS, you may (or may not) know it as the Sun StorageTek Availability Suite. The basic idea is that you can have two storage devices attached to different hosts that stay in sync. This product forms the core of the CDP (no, not that CDP) capabilities of the StorageTek and Sun Unified Storage Server product ranges.

Too bad it is completely and totally useless when using some of the standout features of Sun's current opus, ZFS.

Let's take one feature that shows ZFS in all its brilliance - resilvering. As detailed previously, doing this is going to be very bad for anyone who uses big disks and/or less than 10GbE connectivity between AVS nodes. Jim Dunham responded to this, saying he doesn't see a way of making AVS "smarter" or "ZFS aware". The man makes good points, but both he and the original post's author have totally missed the point.

Sun have done the wrong thing here with AVS:

  • AVS is a paid-for add on for Solaris. Even though AVS is a Sun internal product, it's just as bad as VxFS or Veritas Volume Manager. Way to go, Sun!
  • AVS in OpenSolaris is outdated. Really - why bother?
  • AVS + ZFS just doesn't work, ever. It's a half-hearted effort that provides support for getting your data from one box to another, but no support from the vendor for failover unless you buy some black boxes. Take up religion if you hope to fail back gracefully.
ZFS is transactional and has intelligent operation management at its core. Send those transactional operations across the wire. A device resilvering should never have block rewrites replicated to the passive host and a scrub can happen in parallel to normal file system activities.

Take each of the above operations:

  • A scrub completing with corrections successfully on the active host says nothing about disc consistency on the passive host. Begin the operation on both hosts.
  • Even if you have two dedicated, nerdy and doomed-to-be-virgins-forever system administrators in your colo replacing disks in the active and passive nodes at the same time, there will be a difference between the two new disks as ZFS begins to resilver. Your disks are definitely inconsistant, unless you're using hardware RAID...WHICH MANAGES TO ROB YOU OF MANY OF THE BENFITS OF ZFS. For those in the audience that pledge an answer of "full AVS replication", see above. You fail.

If someone were to give me the means to satisfy my financial obligations and work on only ZFS, I would give you the solution to this problem inside of six months without a problem. Seriously Sun - if your (admittedly brilliant) software engineers can deliver the marvel that is RAID-Z in 599 lines, then I can do it.

For those who didn't read the intro of the blog, yes I am a software engineer and yes I am proficient in C, system programming and mathematics. My assertions are justified.

It is a horrible injustice that the greatest filesystem ever is treated in this way.

Enterprise-level hardware

The enterprise, and hardware designed to service it, is concerned with a few simple things:

  • Visibility - Every piece of the server/switch/router/appliance should be easily monitorable and notify of issues before they become problematic
  • Reliability - Failure is not an option. Everything is redundant.
  • Availability - The hardware should never give reason for reboot - ever.
How horribly far away from this we are. Taking the same three points:

  • Visibility - All of the major "enterprise" hardware vendors ship hardware that falls short. ESX and ESXi on modern boxes is just one example of things going horribly wrong in this regard. Internode's most recent mail server failure is a particularly dark mark on whoever their SAN vendor was - the same can be said for the poor people at WebCentral. I've seen plenty of servers reboot when vendor monitoring and health metrics indicate everything is fine prior, only to see RAM, RAID controller, battery and backplane failures become painfully apparent as the system boots afterward.
  • Reliability - This is an interesting one. Take Dell (please, take them) - the 2850 featured redundant memory and dual RAID controllers. The 2950 does not for some strange reason! The replacement for the 2950-III, the R710 reintroduces these features - thanks Dell!
  • Availability - How can this be so incredibly hard? Foundry, Cisco and Juniper can all give me gear that allows me to upgrade firmware online whilst coordinating BGP, OSPF, IS-IS and MPLS. Take a modern server - firmware for the BIOS, DRAC/iLO controller, RAID controller, IPMI/BMC and the list goes on. Why can't all of these be upgraded and be exposed while the system is online? How hard can it be?
It gets worse. Recently we received a package from Dell containing a firmware update CD for Seagate Barracuda drives. Although we don't use any of the affected drives in production, the included literature was very worrying:
  • Sorry - everything will be offline when you upgrade this firmware.
  • If you've connected the drives in certain ways to our high-end RAID controllers, you'll need to power down and change that before you apply the update.
WTF? Even though Dell is a relative latecomer to the enterprise world, surely they're not that far behind. Even though their blade servers are a pathetic joke compared to everyone else (it's called a redundant midplane - all the cool kids have them).

Tomorrow morning I'm going to ask my boss, the benevolent dictator, for 2 SANs set up in a redundant multipathed configuration for these reasons. I'll also ask for a bunch of VSphere 4 licenses so I can use VMotion to get around these horrible shortcomings.

Sometimes I really hate computers.

Sunday, April 5, 2009

You made the time change

So pay me for my effort, you inconsiderate worthless excuses for human beings.

I write this as I sit here watching systems fail due to various reasons that all stem from the Victorian government changing the demarcation points for daylight saving changes, again.

Here's a brilliant set of ideas:

  • The orbit of the earth isn't changing. Pick a set of dates and stick to it. Let me actually have a life and get some sleep.
  • Accept that daylight saving has no impact whatsoever on power consumption. People will find ways of doing whatever it is they do regardless of the outdoor conditions. Thanks to the state of the economy everywhere, staying inside at home is more attractive than ever.
  • Daylight saving is a stupid idea. If you are so incredibly stupid that you think that a government specified time offset change will make a major impact on your life, society's workings or the "useful" hours available to each person, please move far away from here.
Every time some set of idiots decides to yet again change the dates where the UTC offset changes, IT professionals everywhere have to put in wasted effort. Thanks to tonight's events, I'll spend part of my Sunday writing up incident reports.

Thanks to the Victorian government - you should be forced to compensate me and everyone else who does what I do. You're pathetic and have no business running a small kindergarten, let alone a state. Be ashamed - it's the only appropriate action.


Sunday, March 29, 2009

Windows into society...

...or not....

Craigslist is special. For many values of "special".

Where else can you state that you want:

  • To sell cinderblocks
  • Have a fantasy involving Super Mario Bros on NES
  • Criticise people for almost any inconsequential wrong they've commited against you in a public forum
  • Ask someone to meet up with you for some random encounter under worringly specific circumstances
Seriously - where else can you?

Stay tuned for a series of delightful social experiments!


Thursday, March 12, 2009

I know where your gear is

So you may as well just tell me.

Many ISPs, web hosts, service providers and other connected enterprises make a big deal out of not stating where they host their assets.

Here's some news for all the groups who think in that fashion - it's trivial to figure such things out in the vast majority of cases. Just be honest - it's easier.

Unless of course you're hosted in Fujitsu in Melbourne, in which case we'd really rather not know!

Tuesday, March 10, 2009

Here's an unusual test

In a here-to unrevealed factoid, Null is a guy. This is hardly a surprise, as most geeks are guys.

Every now and then, Null will meet up with someone of the opposite sex (as Null is currently single). Null never actually calls and/or contacts this member of the opposite sex after the meeting though unless they contact Null first.

The reason for this is very simple - people are people. If you subscribe to everything being "the man's job" in terms of follow up then you're probably in need of a bit of introspection. If not, then you're genuinely not interested. Either way is ok, and leads to the same outcome - good for them!

Just go after what you want, and disregard what you don't - it won't hurt.

Rules for the first date

Here's a (possibly welcome) departure. Recommended rules of engagement for a first date from a geek...who has a cat...and two degrees...oh dear.

Don't do the following::

  • Talk about loss and hardship - it'll either scare off your date or create a type of closeness that will become either entrapping or awkward. Just don't.
  • Mention your former partner(s). Creating a yardstick on the first meeting is never good - give the person you've chosen to meet up with at least a bit of a chance!
  • Make jokes about your genitals and/or sexual prowess - if you're making jokes about either of these things one or both are probably worthy of a laugh.
  • Mention your recent tragedies - sympathy and fun are totally separate. See above.
  • Your achievements on WoW, Xbox Live or the PS3 trophy case. If your date is just "something different", then you are just "something different" and should probably stay that way.
  • Detail where it all went wrong - we've all got problems. They'll become apparent in time, and more acceptable once your redeeming qualities and underlying decency (don't snigger) are accepted if things get that far.
  • Profess emotional attachment and need. In fact, don't be needy. Be confident, stupid.
  • Show how better you are than your mate, same sex sibling, date's former defacto/spouse/friend with benefits/miscellaneous object of desire explicitly - provoke a thought process and stop being insecure.
  • Perform feats of physical domination with your date, friends nearby, random strangers and/or the police. You'll end up hurting yourself, sweating profusely, injuring yourself, injuring someone else, getting arrested or soiling your pants and releasing a fragrance.
  • Challenging anyone to a contest involving shots and/or Yagerbombs. What are you -16? This statement is especially relevant if the other party is your date! Who exactly is going to make sure you both get home (no, not both to your home or their home - chances are you'll both be useless if it comes to that).
  • Talking about STDs, even as jokes. See the point about former partners above.
  • Impressing with money by discussion and flaunting. This reeks of desperation and will make your date either feel uncomfortable or obliged - if you want either actuality I pity you.
  • Displaying pictures of family. A photo is worth a thousand words - most of them sound like "taxi" if you fill conversation space in that fashion.
  • Disregard talk and/or pictures of pets - what on earth is wrong with you if you don't like animals!?!
All that said, kissing and sex on the first date is completely and totally fine.

Assumption is easier....

..no matter how vacuous, stupid, inadvisable and negligent.

Assuming that you know the answer is quicker and involves less effort.

If you are in a technical role, assumption allows you to rest on your laurells without resorting to checking. If you are in a managrial role, assumption excuses you from executing your assigned duties.

In either case, you are guilty of commiting an epic fail. I sincerely hope that your immediate superior disapproves of the pornography you're trying to hide from everyone in an extremely lame fashion.

If you are a true professional, you should be dedicated to correctness, completeness and inclusion at all times. Else, you are a hack - please move to the back of the queue.

It is not a sign of weakness, stupidity, admission or surrender. You can disguise it as having authority over a person or group of people, as checking the correctness of other people's work or having your lowers validate your authority. Better yet, you could be truthful and open and state that you want to know what people think so that discussion and debate allows everyone to arrive at an answer facilitated by leveraging and recognising the collective knowledge, experience, intelligence and insight affored to you by the people who have chosen to work with you.

If you're not talking to the people you work with, either move on or get with the program.



Sunday, February 8, 2009

Do less work

It's better for all involved.

All forms of optimisation involve doing less work. There is no debating this.

Quicksort is optimal. Sedgewick showed this mathematically (and his mentor, the great Knuth agreed) in terms of the amount of work done and movement of data elements required.

The rules are simple:

  • Don't poll - let the runtime / OS tell you when there is work to do
  • Respond to events, don't anticipate conditions.
  • Dispatch quickly
  • Decouple and hand off work units - seeing to completion only increases latency and reduces throughput
No number of threads and processors negate these rules. Sorry Sun - Niagra doesn't relieve us of thinking at least somewhat.

Monday, January 19, 2009

Is humanity getting worse...

...or just more thoroughly exposed and understood?

At my workplace, there is a contest. The contest was originally something along the lines of "whoever finds the worst thing on the Internet wins." Of course there was no real prize other than the respect and/or disgust of the other people that worked there.

The contest then devolved into "whoever finds something that makes Null do anything but laugh wins." Changing the contest to this was a bad idea, as it turns out that the contest is now seemingly not able to be won, and I have to play by the old rules. That doesn't really matter though, as I always win.

Getting back to the topic of this post, is it that humanity is getting worse or that we're seeing more of it, thanks partially to the internet. Personally I'm in favour of more exposure as opposed to greater descent.

That said, I'm not quite sure that things have always been quite this "open".

What does everyone else think?