Methodology F

If I ran the ACCC (The Group of Four).....

2011-01-15T01:20:00.003+11:00

For those not in the wonderful country of Australia, we have an odd situation here in terms of IP connectivity.

In most of the world, there are carriers who call themselves "Tier 1" who are deserving of the title - they have world-wide networks, own prodigious amounts of international and domestic capacity and provide services at a reasonable price. These companies achieve such status via success in business - they've become successful by running their companies well and providing products that have captured a market. Even "near Tier 1" carriers have become impressive by the same means.

In Australia, things are a little bit different.

Here we have this delightful monstrosity called The Group of Four. In 1998 the ACCC decided that they would react to various grumbles by other carriers and force four carriers to engage in peering, creating the GoF. for those not interested in Wikipedia's ramblings, the four carriers today are:

Telstra
Optus
AAPT
Verizon Business

As a result of the ACCC's decision over a decade ago, if any other organisation or individual wishes to provide guaranteed access to or from all areas of the internet that terminates within Australia, they have to (directly or indirectly) source connectivity from one of these companies. The impact of this braindead truth is a bit worrying, in a number of ways:

Two very large residential ISPs (iiNet and TPG) provide inferior access to their customers to many domestic destinations. Instead of connecting to a GoF member at each of their state Points of Presence, they instead only provide connectivity in Sydney (and in Perth in the case of iiNet, only because they wanted to satisfy WoW addicts). Obviously this choice is a business decision, not a technical one.
One most excellent carrier (Internode) connects to Optus in almost every state and Verizon in Sydney. Ironically, this can cause inferior access paths to be presented to customers due to the way BGP works. As good as this is for customers in terms of service, it surely drives up Internode's provisioning costs, eating into their profits and resulting in higher prices for customers out of simple business sense.
Telstra and Optus have significantly lower network costs than any other provider of residential services thanks to their significant access network assets coupled with their completely free access to the Australian internet. In every case, a host within Australia is a customer, a customer of one of their three settlement-free peers or a customer of one of the two previous groups of company. AAPT's sale of their residential customer base and Verizon's absence from the residential market allows them to concentrate on the more lucrative business space.

All that said, the title of this post is "If I ran the ACCC...". So, without further ado, I would:

Forcibly dissolve the Group of Four, divulging the details of the agreement brought about in 1998 to the public.
Require that peering at at least one peering point with an MPLA with an opening peering policy that existed on January 1st, 2011 in a given State or Territory be a requirement for any access to the NBN and associated services in said State or Territory.
Require that peering at at least one peering point with an MPLA with an opening peering policy that existed on January 1st, 2011 in a given State or Territory be a requirement for any access to radio spectrum auctions.
Require that peering at at least one peering point with an MPLA with an opening peering policy that existed on January 1st, 2011 in a given State or Territory be a requirement for any awarding of construction contracts for the NBN (yes, I'm looking at you Nextgen Networks).
Require that utilisation graphs for links to MPLA-governed peering internet exchanges be made public by all participating carriers, to ensure that salespeople do not become the solution to congestion problems.

Before anyone thinks I'm mad, let's look at the arguments against my suggestions:

But the GoF will have their businesses impacted - No, they won't. Companies that are stupid enough to buy business services directly from any of these companies will continue to be stupid enough to do so. Residential customers of similar inclination will continue to be similarly inclined. Given the treatment carriers receive when attempting to source services from the GoF, it's quite clear that such business arrangements aren't too valuable to the GoF, and my proposed points have precisely zero impact on non-transit products.
But you're swapping one government-created cartel for another! Equinix, Pipe/TPG and WAIA will be the new Group of Three - If I was to show up and tell David Teoh, Simon Hackett, Michael Malone, Ravi Bhatia and John Linton that they could stop paying Optus, Telstra, Verizon and AAPT for domestic transit, I'm sure they'd provide space in their various PoPs across the country for interconnection free of charge in some sort of non-profit structure. Better yet, we'll just look back in Australia's internet history and ask AARNet nicely to provide a neutral peering option.
Investment in telecommunications will be stalled thanks to large players having less power - yes, these changes that will result in more competition will disincline the government to spend $43 billion, or not.

Of course, if anyone has any counterpoints to my arguments, I would love to hear them.

EMC, IBM, HP, NetApp and Sun - undeserving monopolists

2011-01-03T23:27:00.000+11:00

Storage.

For those of us who use computers, it is the most boring and basic element of involvement.

Adding 200% of the current storage throughput available rarely gives us more than a 10% increase in real-life benefits thanks to applications being more interested in order of operations than scheduling, and operating systems being more interested in scheduling than order of operations.

With the rise and rise (and subsequent fall) of ZFS, the role of the FS to do the right thing in the best way possible without delegating responsibilities to logical volume managers, RAID controllers and/or remote storage targets has been realised (and then discarded).

Thanks to a combination of geeks focusing on issues that are "fun", application developers focusing on issues entirely within their constrained domain and Linux kernel developers ignorantly choosing their targets without any regard for the direct employers and (n + 1) indirect benefactors who make their employment possible (indirect FFS/extX FS inodes - you all know about such things), Linux is in a sad and embarrassing state in this area. DRBD is a great piece of software, but it may very well propagate file system problems as fast as your well-designed (and implemented) network will allow.

Don't take my word for it - there are plenty of people beyond my rambling self who agree.

So, as a direct consequence of this current shortcoming, I am announcing a project (name to be made public in the next week) that will provide architecture and OS independent CDP and near-CDP, with appropriate knobs to allow System Administrators to decide which trade-off they need to allow between performance and availability whilst allowing for implementation of local, near and remote DR in any conceivable combination.

In terms of implementation, the implementation couldn't care less which OS, filesystem or hardware you use. The implementation writes bytes to files as requested - exactly as write() and pwrite() do.

I want to know about writes.... (Linux Kernel Team - EPIC FAIL)

2010-12-28T03:52:00.001+11:00

Yet Linux tells you, in no uncertain terms, to fuck off.

Here's my scenario in a current software project:

Process W attempts to write X bytes to file Y at offset Z
My filter examines said request and based on some specific criteria (that some overworked, under-financed system administrator is mandated to enforce) allows or denies such request
I have no desire to diminish the performance of the host running my software more than is absolutely required - this means that implementing a FUSE solution is bad (userspace->kernelspace->userspace->my software->kernelspace->fs verses userspace->kernelspace->my software->kernelspace continue),

Reading the above, it all looks quite routine. For those of you thinking this, you are unfortunately wrong.

It seems that the Linux kernel team have precisely zero regard for commercial reality. I will happily apologise in the most public arena possible if this is incorrect but the supporting evidence is in my favour:

* Windows has provided this functionality in a standard fashion for 15+ years. Please, please, please don't make me explain why this is a terrible indictment.

* Other UNIXes have offered this functionality in well-defined ways for a decade

* DOS offered such functionality to user-space applications

* Linux Security Modules offered the "hooks" to make this happen, but LSM was let into the kernel without any regard to multiple users of said functionality - the next step was to remove the API.

* grsecurity, RSBAC, Dazuko, RedirFS, McAfee, CA, AVG and Trend Micro are all at a stupid disadvantage with regard to your indifference to reality - access control to file system assets is a modern requirement, and Linux is alone in providing no standardised API for these applications to provide such in a standardised, efficient and simple-to-implement fashion. The teams responsible for OSSEC, Osiris and Tripwire curse you in a similar fashion.

Before the more versed of you respond:

* syscall hooking - yes, I'll totally interfere in the kernel's operations by inserting code not at all verified by the OS vendor. And while I'm at it, I'll disable any software taking a similar approach, or forgo any decently weighty contract of my own software being in effect and disable SELinux, AppArmor and so on. Grow up.

* FUSE - sure, let's engage in 5 copies of data instead of 2. Learn to count.

* Kernel patching - without taking away from the work of DazukoFS (Dazuko and RedirFS are precisely worthless for this application), DazukoFS doesn't actually work on any Linux distribution that anyone who isn't planning to upgrade in the next 4 months actually uses or is supported by any application that any business actually cares about other than Samba. As good as DazukoFS is, it may as well not exist at the moment (see http://dazuko.dnsalias.org/wiki/index.php/Dazuko-based_Applications if you don't believe me).

So, in summary, it's easier to implement anything that worries about the specific operations on files cleanly on Windows than it is on Linux. One is an OS that I love, promote at every possible opportunity, performs better on every modern hardware platform under the sun, provides more out of the box functionality and the other is Microsoft Windows.

Sorry Linux Kernel Team - EPIC FAIL.

Sydney is not the center of the Internet

2010-05-16T22:30:00.000+10:00

Definitely not in Australia, and even more so anywhere else.

Here is a list of ISPs and carriers that have a problem that I shall verbalise on:

Three*
Vodafone*
TPG
SuperNerd*
iiNet
Virgin*
Nextgen Networks*
Digital River*

The list could go on, but there is no need.

All of these providers have the following problem - all of their transit is in Sydney. Not their international transit, all of it. The starred entries fail additionally, as they peer poorly, if at all.

Traffic from any of these providers for a Melbourne-terminated connection will be tromboned to Sydney, out to the internet and then back again. In the case of the unstarred entries, they do much better in that they peer at Pipe's Melbourne IX and announce at least part of their network there. This alleviates the problem in many cases, but not all.

For example, my home ADSL2+ connection is provided by Mr Teoh's elves from TPG. Connecting to gear that uses connectivity from those miserable proles in the Group of Four results in this delightful tromboning I mentioned above. The same is true of iiNet, and iiNet only have a single PoP in Melbourne so they're ironically spending less on their core infrastructure than TPG is.

The point is that the captive nature of the Australian transit market results in such patterns emerging for savvy businesses such as TPG and iiNet - peer everywhere and use transit where needed. As for the other entries, there is another explanation.

They're inept, each and every one of them. Failing to peer in Melbourne, Brisbane or Perth when you have a local Point of Presence AND relying on intercaptial backhaul for all connectivity is just braindead. SuperNerd are extremely guilty of this, as are Nextgen and Virgin. There is no excuse whatsoever to not pay Pipe the comparatively tiny sum of $1000/month for 100Mb/s peering in Melbourne and Brisbane and even less excuse to not connect to WAIX in Perth.

TPG's stance is almost understandable, given their pricing and their incredibly lazy approach to their usage of their SX/PPC-1 bandwidth, but iiNet's stance is a bit more puzzling. They have a large number of customers on-net, they're the 3rd largest ISP by customers in Australia, have managed to get a price on Telstra+Reach transit that makes sense commercially yet still refuse to terminate domestic transit anywhere other than Sydney. Fail!

Tier 1 Servers, or not

2010-05-16T21:43:00.000+10:00

At work I always find myself buying stuff to make other stuff possible.

Recently we needed to make a small purchase of about $80k to complete a HA project I've been working on for a while. We decided to ask for quotes from both Dell and an IBM partner for this project, as we've been looking at diversifying our hardware base for a while now.

For some reason that doesn't have any bearing to this story, a company had been harassing one of my coworkers for the opportunity to quote on our hardware needs. They insisted they could save us money and replace our Dell servers with "Tier 1 hardware". This leads to a number of questions.

What is Tier 1 hardware?
Why do I want it?
How much is it worth?

The answers to these questions seem to be:

Anything not made by Dell
Because some sales person with a target for the month says so
More than any sane person should pay

For reasons I am yet to fathom, two quotes were provided by said company. Their quotation for the HP gear was around $90k and their quotation for the IBM gear was around $185k. So much for the saving money part of the bargain.

I find it hard to believe that IBM would actually charge such a premium for their gear. Turns out they don't, as a better specified quotation came in from our preferred IBM partner at around $80k. I've no idea where to start in figuring out how such things happen.

Regardless, I relayed this information back to the all-promising vendor and was told that they'd go back to HP to get a lower quote. They did so, sort of.

In the end I didn't bring the HP quotation to the boss as it was clear that I was getting a bad deal in many ways. The hardware on offer featured higher prices, a lack of integrated management, fewer NICs and a complete lack of redundant and hot-swappable cooling.

I sent a very to-the-point message to the all-promising-supplier after the fact, quoted verbatim here:

Hi Supplier-Representative-7G,

I've spoken with my MD and he's looked at the quotations we received from Dell, yourself and another IBM partner.

For a number of reasons, such as the HP servers lacking redundant cooling and being the most expensive options by far, we've opted not to engage All-Promising-Supplier as a hardware supplier.

Best regards,

Methodology F

Yet, my phone rings at regular intervals, displaying the number of Supplier-Representative-7G. I do not answer these calls, yet still they come.

The next lesson shall be "Tier 1 connectivity, or not"

NBN - get a clue

2010-05-07T23:31:00.002+10:00

I live in Australia. The current Labour government is very limited in their outlook on certain issues.

There is much talk of filtering the web and building out fibre. For fuck's sake - get a clue.

Regarding filtering, anyone with 3 braincells and/or a slight want for dissent can easily circumvent any measure you're proposing to put in place. As most of your constituency are bogans, they'll engage in circumvention not for any reason that serves their own purposes, but because their mate Robbo told them about such possibilities at the pub whilst flooding his gullet with Vaginal Backwash.

There is no mention of IPv6, no mention of resiliency and no statement of application.

What will the average Australian moron do with a 100Mb/s link to the rest of the world? Read poorly composed e-mail from his mate Johnno? Watch the latest YouTube viral in 1080p resolution? Upload pornography featuring his girlfriend/wife to some amateur site or transmit the same to that brilliant bloke ORSM?

The vast majority of Australia's population has no idea how their IP feed is provided. Most of them still refer to such connectivity as "my internet" because they're ignorant of the world they live in and are more comfortable with cars, darts and getting into fights at the local pub that considers Victoria Bitter, Carlton Draft and Toohey's New to be a diverse beer selection and get quite agitated if someone dares remove Today Tonight or A Current Affair from the ludicrously large screen whilst they're engaging in drunken escapism between their drudgery of employment and their unhappy de-facto+spawn home life.

Idiot bogans make up a worrying proportion of our populace. The outcome of my ramblings can be summarised thusly:

The bogan wants faster. Not because they want or need it, simply because it is available.
The bogan demands lower prices. It does not matter if their provider of choice is making $1.00 of profit per month, they want a lower price.
The bogan insists that it is being ripped off and that all decisions of their chosen service provider are designed to disadvantage said bogan. A change to any terms or conditions attached to the bogan's service are clearly a personal attack on the bogan.

It clearly follows that all home IP services in my fair country are designed to snare and placate the bogan. Thus, bogans are the problem.

Greatness

2010-05-07T23:08:00.002+10:00

Early in my career, an RMIT professor asked me which I should pursue of two choices - achievement or greatness. My answer was greatness, and he chastised me in a most eloquent way.

How appropriate for a man of age 52 to belittle a child of 19 on issues that he'd spent his career examining and lecturing on. Unfortunately my memory is better than his, and my current thinking allows me to respond to his shallow and morally-determined outlook on the path of the individual.

C, Cobol, VAX, TCP/IP, Cisco, Linux, C++, Java, .Net, Cray, Amdahl, The Art, Google, Microsoft

All of these companies, standards and languages have one simple thing in common - greatness.

A limited group of people was involved in each, and their names are synonymous with their creations and legacy. They are identified for their contribution to the world we live in and the communities they created.

Achievement is a very loose and qualitative term from one point of view. A couple's otherwise unremarkable child has "achieved" by getting the best score in the class on a mathematics test or appearing in the school play. As much as my writing will offend some, such outcomes have precisely zero impact on the limited and wider community in general.

In every single case I have cited, the persons involved aspired to something greater. None of them sought the outcomes of fame, recognition or authority - each had their own particular goals that they achieved.

There are many other groups that have aspired and have not been recognised, but there are examples of this that have given rise to better ideas, more concrete outcomes and further-reaching initiatives. The contributions of such individuals cannot be disregarded.

To the professor who shouted me down almost a decade ago: inferiority - what's it like?

The media is awful

2009-09-14T00:39:00.003+10:00

This evening, I watched Fighting - a movie of questionable utility.

The lead actress was, as is always the case in movies, seemingly perfect in every respect physically. Slim, curvy, attractive and well spoken.

Yet, in the movie, she was earning $11.00 an hour, and supporting her daughter and grandmother. At no point was there any indication that she ate particularly well, worked out or slept enough to remain healthy.

How incredibly beneficial it is to look to mediums of entertainment for reflections on society.

Or not.

PacNet are retarded (and so are AINS)

2009-08-25T01:26:00.003+10:00

PacNet, the strange and no-more-relevant merger of AsiaNetcom and Pacific Internet (at least, it's of no relevance here in Australia), are retarded.

Looking at Pipe's website, PacNet's Sydney DC is a "Pipe Approved" DC. In theory, this means they peer with Pipe.

Taking an IP in their Sydney DC (say 61.14.142.1) and shoving it in Pipe's "Am I Connected?" tool yields most interesting results.

The Melbourne, Adelaide and Brisbane IX points show the IP as reachable, but NSW does not. Which idiot decided to stop announcing that subnet at Pipe NSW? They even announce it at WAIX.

Take iiNet and Internode - their networks will telegraph the traffic to a peering point they can reach in another city for their Sydney users, and then PacNet will carry the traffic back to Sydney!

PacNet are definitely connected to Pipe in Sydney, as their looking glass gets back onto iiNet's network via Pipe NSW. PacNet peer directly with Internode, but Matthew Moyle-Croft (who answers questions about Internode's network on Whirlpool) has made some very "interesting" statements about PacNet's policies since the merger, and the resultant value of the peering relationship.

Why anyone would use their transit capacity or intercapital backhaul when perfectly good peering options are available at a much lower cost is beyond me. The only other carrier/provider I know of that does this are those inept fools at AINS/SuperNerd. That said, they did manage to get one of their peers to announce one of their subnets today but not route the traffic for some reason I'm yet to be informed of.

Thanks to the internet....

2009-07-07T23:07:00.003+10:00

I have learned the following, obviously true things:

I am obese, almost morbidly so (thanks, National Heart Lung and Blood Institute!)
Milk is the answer to getting thin, and will also cause large amounts of fat gain (thanks, BodyBuilding.com!)
My legs hurt when I run because of bad shoes, overtraining, sodium deficiency, too much sodium, or one of 5 different STIs (thanks, Wikipedia and the Better Health Channel!)
The best way to get from Alaska to Queensland is in a kayak....
I can empower myself as a man by making a few modifications.
The pot at the end of the rainbow isn't always filled with gold.
Now that I know I'm obese, I may as well get some support in getting to can't-leave-the-house-fat.
I'm apparently Jewish, lactating and keep a bloke locked inside a box in my appartment.

Two words. Epic Fail.

AVS is a really bad idea

2009-05-31T19:47:00.003+10:00

For those not familiar with AVS, you may (or may not) know it as the Sun StorageTek Availability Suite. The basic idea is that you can have two storage devices attached to different hosts that stay in sync. This product forms the core of the CDP (no, not that CDP) capabilities of the StorageTek and Sun Unified Storage Server product ranges.

Too bad it is completely and totally useless when using some of the standout features of Sun's current opus, ZFS.

Let's take one feature that shows ZFS in all its brilliance - resilvering. As detailed previously, doing this is going to be very bad for anyone who uses big disks and/or less than 10GbE connectivity between AVS nodes. Jim Dunham responded to this, saying he doesn't see a way of making AVS "smarter" or "ZFS aware". The man makes good points, but both he and the original post's author have totally missed the point.

Sun have done the wrong thing here with AVS:

AVS is a paid-for add on for Solaris. Even though AVS is a Sun internal product, it's just as bad as VxFS or Veritas Volume Manager. Way to go, Sun!
AVS in OpenSolaris is outdated. Really - why bother?
AVS + ZFS just doesn't work, ever. It's a half-hearted effort that provides support for getting your data from one box to another, but no support from the vendor for failover unless you buy some black boxes. Take up religion if you hope to fail back gracefully.

ZFS is transactional and has intelligent operation management at its core. Send those transactional operations across the wire. A device resilvering should never have block rewrites replicated to the passive host and a scrub can happen in parallel to normal file system activities.

Take each of the above operations:

A scrub completing with corrections successfully on the active host says nothing about disc consistency on the passive host. Begin the operation on both hosts.
Even if you have two dedicated, nerdy and doomed-to-be-virgins-forever system administrators in your colo replacing disks in the active and passive nodes at the same time, there will be a difference between the two new disks as ZFS begins to resilver. Your disks are definitely inconsistant, unless you're using hardware RAID...WHICH MANAGES TO ROB YOU OF MANY OF THE BENFITS OF ZFS. For those in the audience that pledge an answer of "full AVS replication", see above. You fail.

If someone were to give me the means to satisfy my financial obligations and work on only ZFS, I would give you the solution to this problem inside of six months without a problem. Seriously Sun - if your (admittedly brilliant) software engineers can deliver the marvel that is RAID-Z in 599 lines, then I can do it.

For those who didn't read the intro of the blog, yes I am a software engineer and yes I am proficient in C, system programming and mathematics. My assertions are justified.

It is a horrible injustice that the greatest filesystem ever is treated in this way.

Enterprise-level hardware

2009-05-31T19:24:00.002+10:00

The enterprise, and hardware designed to service it, is concerned with a few simple things:

Visibility - Every piece of the server/switch/router/appliance should be easily monitorable and notify of issues before they become problematic
Reliability - Failure is not an option. Everything is redundant.
Availability - The hardware should never give reason for reboot - ever.

How horribly far away from this we are. Taking the same three points:

Visibility - All of the major "enterprise" hardware vendors ship hardware that falls short. ESX and ESXi on modern boxes is just one example of things going horribly wrong in this regard. Internode's most recent mail server failure is a particularly dark mark on whoever their SAN vendor was - the same can be said for the poor people at WebCentral. I've seen plenty of servers reboot when vendor monitoring and health metrics indicate everything is fine prior, only to see RAM, RAID controller, battery and backplane failures become painfully apparent as the system boots afterward.
Reliability - This is an interesting one. Take Dell (please, take them) - the 2850 featured redundant memory and dual RAID controllers. The 2950 does not for some strange reason! The replacement for the 2950-III, the R710 reintroduces these features - thanks Dell!
Availability - How can this be so incredibly hard? Foundry, Cisco and Juniper can all give me gear that allows me to upgrade firmware online whilst coordinating BGP, OSPF, IS-IS and MPLS. Take a modern server - firmware for the BIOS, DRAC/iLO controller, RAID controller, IPMI/BMC and the list goes on. Why can't all of these be upgraded and be exposed while the system is online? How hard can it be?

It gets worse. Recently we received a package from Dell containing a firmware update CD for Seagate Barracuda drives. Although we don't use any of the affected drives in production, the included literature was very worrying:

Sorry - everything will be offline when you upgrade this firmware.
If you've connected the drives in certain ways to our high-end RAID controllers, you'll need to power down and change that before you apply the update.

WTF? Even though Dell is a relative latecomer to the enterprise world, surely they're not that far behind. Even though their blade servers are a pathetic joke compared to everyone else (it's called a redundant midplane - all the cool kids have them).

Tomorrow morning I'm going to ask my boss, the benevolent dictator, for 2 SANs set up in a redundant multipathed configuration for these reasons. I'll also ask for a bunch of VSphere 4 licenses so I can use VMotion to get around these horrible shortcomings.

Sometimes I really hate computers.

You made the time change

2009-04-05T02:50:00.002+11:00

So pay me for my effort, you inconsiderate worthless excuses for human beings.

I write this as I sit here watching systems fail due to various reasons that all stem from the Victorian government changing the demarcation points for daylight saving changes, again.

Here's a brilliant set of ideas:

The orbit of the earth isn't changing. Pick a set of dates and stick to it. Let me actually have a life and get some sleep.
Accept that daylight saving has no impact whatsoever on power consumption. People will find ways of doing whatever it is they do regardless of the outdoor conditions. Thanks to the state of the economy everywhere, staying inside at home is more attractive than ever.
Daylight saving is a stupid idea. If you are so incredibly stupid that you think that a government specified time offset change will make a major impact on your life, society's workings or the "useful" hours available to each person, please move far away from here.

Every time some set of idiots decides to yet again change the dates where the UTC offset changes, IT professionals everywhere have to put in wasted effort. Thanks to tonight's events, I'll spend part of my Sunday writing up incident reports.

Thanks to the Victorian government - you should be forced to compensate me and everyone else who does what I do. You're pathetic and have no business running a small kindergarten, let alone a state. Be ashamed - it's the only appropriate action.

Windows into society...

2009-03-29T21:11:00.003+11:00

...or not....

Craigslist is special. For many values of "special".

Where else can you state that you want:

To sell cinderblocks
Have a fantasy involving Super Mario Bros on NES
Criticise people for almost any inconsequential wrong they've commited against you in a public forum
Ask someone to meet up with you for some random encounter under worringly specific circumstances

Seriously - where else can you?

Stay tuned for a series of delightful social experiments!

I know where your gear is

2009-03-12T01:05:00.002+11:00

So you may as well just tell me.

Many ISPs, web hosts, service providers and other connected enterprises make a big deal out of not stating where they host their assets.

Here's some news for all the groups who think in that fashion - it's trivial to figure such things out in the vast majority of cases. Just be honest - it's easier.

Unless of course you're hosted in Fujitsu in Melbourne, in which case we'd really rather not know!

Here's an unusual test

2009-03-10T01:25:00.002+11:00

In a here-to unrevealed factoid, Null is a guy. This is hardly a surprise, as most geeks are guys.

Every now and then, Null will meet up with someone of the opposite sex (as Null is currently single). Null never actually calls and/or contacts this member of the opposite sex after the meeting though unless they contact Null first.

The reason for this is very simple - people are people. If you subscribe to everything being "the man's job" in terms of follow up then you're probably in need of a bit of introspection. If not, then you're genuinely not interested. Either way is ok, and leads to the same outcome - good for them!

Just go after what you want, and disregard what you don't - it won't hurt.

Rules for the first date

2009-03-10T00:49:00.003+11:00

Here's a (possibly welcome) departure. Recommended rules of engagement for a first date from a geek...who has a cat...and two degrees...oh dear.

Don't do the following::

Talk about loss and hardship - it'll either scare off your date or create a type of closeness that will become either entrapping or awkward. Just don't.
Mention your former partner(s). Creating a yardstick on the first meeting is never good - give the person you've chosen to meet up with at least a bit of a chance!
Make jokes about your genitals and/or sexual prowess - if you're making jokes about either of these things one or both are probably worthy of a laugh.
Mention your recent tragedies - sympathy and fun are totally separate. See above.
Your achievements on WoW, Xbox Live or the PS3 trophy case. If your date is just "something different", then you are just "something different" and should probably stay that way.
Detail where it all went wrong - we've all got problems. They'll become apparent in time, and more acceptable once your redeeming qualities and underlying decency (don't snigger) are accepted if things get that far.
Profess emotional attachment and need. In fact, don't be needy. Be confident, stupid.
Show how better you are than your mate, same sex sibling, date's former defacto/spouse/friend with benefits/miscellaneous object of desire explicitly - provoke a thought process and stop being insecure.
Perform feats of physical domination with your date, friends nearby, random strangers and/or the police. You'll end up hurting yourself, sweating profusely, injuring yourself, injuring someone else, getting arrested or soiling your pants and releasing a fragrance.
Challenging anyone to a contest involving shots and/or Yagerbombs. What are you -16? This statement is especially relevant if the other party is your date! Who exactly is going to make sure you both get home (no, not both to your home or their home - chances are you'll both be useless if it comes to that).
Talking about STDs, even as jokes. See the point about former partners above.
Impressing with money by discussion and flaunting. This reeks of desperation and will make your date either feel uncomfortable or obliged - if you want either actuality I pity you.
Displaying pictures of family. A photo is worth a thousand words - most of them sound like "taxi" if you fill conversation space in that fashion.
Disregard talk and/or pictures of pets - what on earth is wrong with you if you don't like animals!?!

All that said, kissing and sex on the first date is completely and totally fine.

Assumption is easier....

2009-03-10T00:38:00.002+11:00

..no matter how vacuous, stupid, inadvisable and negligent.

Assuming that you know the answer is quicker and involves less effort.

If you are in a technical role, assumption allows you to rest on your laurells without resorting to checking. If you are in a managrial role, assumption excuses you from executing your assigned duties.

In either case, you are guilty of commiting an epic fail. I sincerely hope that your immediate superior disapproves of the pornography you're trying to hide from everyone in an extremely lame fashion.

If you are a true professional, you should be dedicated to correctness, completeness and inclusion at all times. Else, you are a hack - please move to the back of the queue.

It is not a sign of weakness, stupidity, admission or surrender. You can disguise it as having authority over a person or group of people, as checking the correctness of other people's work or having your lowers validate your authority. Better yet, you could be truthful and open and state that you want to know what people think so that discussion and debate allows everyone to arrive at an answer facilitated by leveraging and recognising the collective knowledge, experience, intelligence and insight affored to you by the people who have chosen to work with you.

If you're not talking to the people you work with, either move on or get with the program.

Do less work

2009-02-08T02:25:00.002+11:00

It's better for all involved.

All forms of optimisation involve doing less work. There is no debating this.

Quicksort is optimal. Sedgewick showed this mathematically (and his mentor, the great Knuth agreed) in terms of the amount of work done and movement of data elements required.

The rules are simple:

Don't poll - let the runtime / OS tell you when there is work to do
Respond to events, don't anticipate conditions.
Dispatch quickly
Decouple and hand off work units - seeing to completion only increases latency and reduces throughput

No number of threads and processors negate these rules. Sorry Sun - Niagra doesn't relieve us of thinking at least somewhat.

Is humanity getting worse...

2009-01-19T00:48:00.002+11:00

...or just more thoroughly exposed and understood?

At my workplace, there is a contest. The contest was originally something along the lines of "whoever finds the worst thing on the Internet wins." Of course there was no real prize other than the respect and/or disgust of the other people that worked there.

The contest then devolved into "whoever finds something that makes Null do anything but laugh wins." Changing the contest to this was a bad idea, as it turns out that the contest is now seemingly not able to be won, and I have to play by the old rules. That doesn't really matter though, as I always win.

Getting back to the topic of this post, is it that humanity is getting worse or that we're seeing more of it, thanks partially to the internet. Personally I'm in favour of more exposure as opposed to greater descent.

That said, I'm not quite sure that things have always been quite this "open".

What does everyone else think?

Some knowledge is best withheld.

2009-01-07T00:22:00.002+11:00

Really, it is.

Take this delightful example:

Opening the Windows Vista Box

There are few facts that could possibly facilitate the necessity of this information existing:

The box is incredibly difficult to open, thus needing instructions. This indicates a terrible design flaw which should be well outside of Microsoft's domain of folly (stop sniggering all of you in the back)
You suspect the people likely to use your product are completely and totally mentally vacant.

Either way you've insulted the people tasked with opening the box. You've done everyone a grave disservice.

On the same level, it is often not in your interest to give complete details to people you task with getting something technical achieved. From first principals comes understanding and many other useful things.

A non-technical person (they call themselves that - they don't let on how much they know) and a very scientifically minded person (who has also written code, administered systems and brewed their own beer) where I work subscribe to this idea in their own way.

The non-technical person favours provoking thought over prescribing action. The scientific guy favours proving you are correct as opposed to following a prescribed recipe. I'm slightly different again, but have adopted parts of both approaches.

In the theme of being profound through brevity:

Listen - Learn - Think - Question

It's rare someone has nothing to teach you, so listen. It's rare that someone's viewpoint entirely aligns with yours, so think and consider. And it's also rare (in professional life) that you have nothing to contribute, so question.

Simple really.

The "art" of being "right"

2009-01-06T00:36:00.003+11:00

And, while we're at it:

RIGHT != CORRECT

For the purposes of this post at least!

Right - acknowledged as the correct viewpoint; accepted as correct by other parties; articulation of popular opinion.
Correct - a strongly founded standpoint derived from evidence, logical deduction and process; undeniably applicable and unchallengeable by any.

Better definitions of the above are welcomed.

The two are different for a few reasons. For those familiar with the history of our artform and industry, the "correct" option has not always won. The Amiga was superior to the Macintosh when both were in the marketplace, MCI was superior to EISA, NetWare was superior to Windows NT and IPX/SPX (and AppleTalk for that matter) was superior to TCP/IP. All these statements are true on a purely technical level, but the outcomes would have the casual observer conclude otherwise.

In all cases, the surviving technologies above were right, if not at all correct. The Macintosh won due to software support and marketing, EISA won due to backward compatibility and ubiquity, Windows NT won due to marketing and identifiability and TCP/IP won due to existing vendor support (and AppleTalk for similar reasons). The surviving solutions were all right though.

In our industry, being right will serve you better than being correct (if it is a choice between the two). The correct solution tends to involve more time, effort and dedication than less technically-inclined and passionate stakeholders are willing to accept and tolerate. Our dedication to skill set and furtherment are tempered by others - it is the reality.

Which brings me to the "art" of being "right". In the words of my benevolent dictator who is almost always right "it isn't that hard":

Be undeniably clear - Leave no fact or detail up to interpretation. Expound, explain and clarify. Leave no room for debate or deviation.
Be correct in your use of language - Do not misuse terms. The moment you do, the nerd in the corner can unravel your otherwise well-weaved fabric of assertion and logic. Be prepared to define and explain.
Concede as quickly as possible - Harder than it seems. The moment you need to concede, you may need to rethink a large amount of what you have left to say or have already said. It is much better to show that you're in command of the subject matter in the face of challenge instead of holding steadfast to belligerent malfeasance.
Tell people they are wrong - You're doing them a favour if they're dedicated to learning, correctness, doing their job properly or being remotely professional. They're not lying, they're presenting facts as they know them. Just be prepared to support your point in a fashion that focuses on facts as opposed to personal failing.
Stay focused - Make a point, not a stand. You're not there to witness the pontification of others, as that doesn't serve anyone (see point 3).
Listen and respond - don't hear and disregard - If you have surrounded yourself with people who are one of Yes Men, people you consider your lessors or those with nothing to contribute no amount of reading or thought can possibly help you. Thus you are either beyond help or surrounded by people who are there to help you and are more than capable of doing so. Leverage expertise and have your expertise leveraged.
Challenge - don't quash - Being prepared to challenge and demeaning a person are two different and incompatible actions. The challenge improves all involved - being quashed improves no one.
Be deliberate and thorough in accepting concession - Someone accepting your viewpoint as correct and vacating their own is no different to the reverse occurring. Confirm why they see your assertions are now correct in their view.

The strange thing with this points is that being correct will not require these rules to be followed when in the company of the faithful - they will convince themselves.

What is wrong with people!?!?

2008-12-27T01:15:00.002+11:00

Honesty is seemingly core to almost every society. Seemingly...

Of course not everyone learns the lessons society seeks to teach them. Arrogance, ignorance, belligerence and emotional depravity all detract from the learning process.

How hard is it to tell someone that they've acted in the wrong way - a fashion inappropriate given the community in which they lead their lives? It's seemingly impossible without being branded harsh, prickish, inconsiderate, lacking in compassion, insular or cruel.

As parents such enforcement is often associated with being overly disciplinarian, as an employer totalitarian and selfish if done as a friend.

Every know and then someone needs to be dealt with harshly for their own benefit. Unfortunately, even if the person dealing out the seemingly harsh treatment is doing so for reasons more personally invested than professional duty, the true intent is very rarely recognised.

Some people really, really need to wake up and take a good look at themselves.

Joel is wrong

2008-12-21T23:40:00.002+11:00

Like many of you (I think, and possibly hope), I've read Joel of Fog Creek Software's blog on occassion.

The man tends to be correct. Every now and then though, the man is just completely and totally wrong.

I remember a blog post where he advocates hiring the best programmers exclusively. What on earth is wrong with the man!?

He's wrong on two levels:

Hiring only "the best" creates a terrible culture of elitism and exclusivity. Do you really want to make the cost of hiring escallate that much as time goes on?
"The best" are, by definition, an extremely finite resource. You want to cut down your potential workforce and then make it all the more difficult to integrate them?

Good grief. These two facts are actually the lesser of your problems.

Let's assume you get a team of 4 brilliant developers who write amazing code, backed by perfect design. Your problems have begun.

Over the next 5 years, all 4 of your developers move on. Face it, they're "the best" - if you've only got 4 developers what are the chances you can keep them interested? What happens when you replace them? Are the replacements going to be equiped to handle whatever they produced in terms of maintanance? Will they be able to support your sales staff, support staff, technical writers, strategists and other stakeholders?

These are questions that need to be asked - Joel doesn't seem to cover them all that thoroughly at all.

Logs are an admin's best friend....

2008-12-21T23:15:00.002+11:00

Or their absolute worst enemy, depending on what's producing them.

One part of my employer's network needs to satisfy one of the industry's most intense and invasive standards - the Payment Card Industry Data Security Standard. Many organisations are grappling with gaining compliance at the moment - it's not at all easy.

One of the major parts of compliance (and an aspect that our QSA identifies as a regular failing of organisations aiming to achieve compliance) is logging. Specifically the retention of logs and protection of logs.

Proven, standardised solutions work best so we of course deploy syslog servers to capture the huge log traffic from the myriad of devices we deploy to obtain our tick each year.

Last Tuesday morning at 0:15 I was about to push some scheduled firewall changes to one of our production firewall clusters. As I'm not a complete and utter nutter (commenters may disagree), I was at home at the time and doing this remotely.

All attempts to bring up either the SSH or web interface to the firewall cluster failed. Logging in to the office VPN and trying from there failed. I could get into our production network without issue but even from there, getting to the firewall simply wasn't happening. Down to one of Melbourne's tier 3 data centers I went.

At no time was there any indication of any network slowdown at that time, and all our monitoring showed no issues at all. I managed to get to the console on the then active firewall and couldn't see anything standing out as being a problem but the CPU usage was 5 times normal - never good, but at least I had an explaination for the lack of reachability.

Given these lovely devices are clusted together and no indication otherwise existed, I suspected some weird software fault and forced the other node of the cluster to take over. Access to the main administrative interface was restored...for about 75 seconds. The high CPU usage returned and the SSH and web access dropped away. Our SNMP monitoring was still working, and showing no increase in session count, attacks or memory usage - just high CPU usage. Failing back didn't help either.

After an awful lot of buggering around, I managed to trace the problem to one of our syslog servers. A quick reboot fixed the problem.

Logs were the cause of the problem, and there was no indication anywhere that the syslog server was at fault - we use Juniper gear and the syslog connections did not appear in the output of "get session". This is the second time we've discovered a "silent failure" on our Juniper devices.

I've raised this with Juniper. I suspect that I'll never get a solid resolution - just like we didn't last time.

Tolerance above all else

2008-12-21T22:57:00.003+11:00

For my bretheren, you already know the following. For those who we report to (and pay us - we have the utmost respect for you, really), please take note:

* All processes fail

* Something is always missed

* Failures happen at the worst possible time

The corollary of this is simple, obvious and an instructive bastardisation of an oft quoted phrase:

One must plan to fail

Ironically it is the best admins, developers and DBAs that tend to understand and work with this particular factoid. The ones who have gotten well past the stage of "my work is perfect" and "not my problem".

There are developers, admins and DBAs much more experienced and older than myself who fail to embrace this simple idea. We have a long, long way to go.

There are no marbles

2008-10-26T05:16:00.002+11:00

Recently my boss asked me how I ensured that two database servers contained the same data.

The two servers in question ran Microsoft SQL Server 2000 (don't hate me - I inheritted them) and the only replication option available was Log Shipping.

Those who have had to go through this process will know that a log either applies fully or does not. Microsoft SQL Server 2005 (and Sybase ASE 15) are more generous - they allow you to apply a previously applied log without error. Regardless, a hard error occurs if a log is applied that does not pick up where the previous log application left off.

If change # 13559 was the last change in the previously applied log, the current log better start with change # 13560, or else the system administrator is going to be alerted / woken up / distracted from drinking / stopped mid-coutis.

Unfortunately, most IT professionals have to deal with managers who have limited (putting it kindly) knowledge of software in general, let alone enterprise systems. Upon being asked about our database failover status, my description was met with the most interesting statement:

"If I want to ensure that two buckets have the same number of marbles, I need to count the number of marbles in each bucket."

The sillines of this statement is left as an exercise to the reader.

Basics - 3

2008-10-26T04:50:00.002+11:00

The greatest error you can make is to forget your earliest lessons.

I'm not referring to forgetting the detail - I write code most days and I haven't written a fundamental algorithm method/function/predicate in years, but the detail can be the thing that holds us back and isn't relevant here.

Whether you're a system admin, software developer or some other form of IT slave, you are dealing with other people's software. Even if you're an OS/compiler/VM/embedded developer, you are dealing with other people's code. Given such preconditions, none of us are special - we've all been taught the same basic lessons (we hope - surely no one is using a linear search in their performance-critical code...oh wait....).

Concepts are the critical thing here - applying operations on any form of efficient data structure tends to eliminate said efficenices, remembering that sorts depend on data size, that searches depend on the structure of the input data and minimising structure size is always in your best interest unless there is some other imperitive that is far more pressing (of which I'm yet to see an example).

Why then I still see queries like this in production software is beyond me:

SELECT * FROM some_table AS t1 INNER JOIN some_other_table AS t2 ON RIGHT(t1.field1, 3) = RIGHT(t2.field1, 3) WHERE LEFT(t1.field2, 3) = 'XYZ' AND LEFT(t2.field3, 3) = 'ABC') ORDER BY t1.field4 ASC LIMIT 0,2001

(No, it's not a MySQL or PostgreSQL database - MySQL would not cope given it's 1-index-per-table-in-most-cases evaluation and PostgreSQL would have filled its logs with rewrite-your-query-already-you-idiot - the query has been recomposed to protect the guilty).

As soon as I spoke to the person who wrote this query, they were very interested in what they did wrong and how to fix the problem (even though no performance isues were evident). Since then they have always checked their work and have done the perfectly right thing in terms of getting queries, code and other artefacts dead right.

Next time you're called a geek by your fellow code warriors or system grunts, remember that you're probably going to be the person who finds the cause of the next big issue. We're all taught the fundamentals early on for a reason. The fact that so many engineering and mathematics students perform poorly on basic operations of addition and multiplication (and their associated inverses) betrays the problem seen in industry here - don't move with the herd.

We are taught be be lazy

2008-10-26T04:36:00.003+11:00

So it makes sense we do less work.

A recent post on The Daily WTF reminded me of something. Specifically it reminded me about the most basic rule of making things happen more quickly.

The particular problem is unlikely to be encountered by anyone building the vast majority of conceivable software thingits, but that doesn't matter - the example is still instructive.

For those not following the link, the basic problem is to find the lowest common multiple of a set of contiguous integers from 1 to X inclusive. The specific case set X = 20.

The brute force solution (as opposed to the "I'm a moron" solution presented in the link) is to try every integer from 1 onwards. For all those thinking that such an approach is acceptable, think again.

Seriously, you have two important facts in front of you:

* The answer has to be evenly divisable by the largest integer in the range presented (we'll call this value Y).

* The answer has to be evenly divisable by the largest prime in the range presented (we'll call this value Z).

Taking the linked example, the largest number is 20, and the largest prime is 19.

So the lowest possible value is the largest integer in the range multiplied by the largest prime in the range - 20 * 19 = 380. The smallest step in the check is 19 - the largest prime. For anyone who queries this, please feel free to leave comments - I like comments and I like maths (yes, I am weird and insane).

From such a simple observation you can easily go further, but a large reduction in the search space has already been achieved with very little effort. In terms of return as a function of effort, a local maxima has most probably been observed.

The moral of the diatribe is similarly simple - the basis of all optimisation is to do less work on a nett basis. The old school world of precalculating sine and cosine values of angles is long past, as is designing systems around binary values, but reusing values and avoiding duplication of work and un-needed work is just as valid as ever.

Basics - 2

2008-07-14T01:04:00.002+10:00

For those in our industry, know this - we stradle many disciplines, savour your place.

For those who work with us - as detailed as we need to be, detail is often the problem.

As ironic as it is, many of my greatest insights have come as the result of a non-technical person speaking their mind.

I'm sure a lot of people in our industry have found that as they've gained more experience, they've spent more time focusing on the basics and finding solutions at that end of the spectrum.

As this is a basic exercise, the message and justification is basic. Just because someone hasn't gone through the same education and experience as you doesn't relegate their input to the level of uselessness. In fact, the exact opposite may be true.

malloc, free, repeat, repeat ad infinitum

2008-07-13T23:42:00.005+10:00

In the current development climate, you'll be using one of two types of development environments:

An environment where the developer is responsible for allocating and freeing resources
An environment where the developer acquires resources, and (generally) the environment takes care of freeing them

Of course, it's not quite so clear-cut. Maintaining a static (or Shared for you VB.Net developers) list of allocated resources will stop any sort of reclaimation from happening, but the distinction is sufficient for this article.

Recently, I had to deal with a delightful issue involving a memory leak on every developer's favourite OS, Windows NT (2000, 2003, XP, Vista, 2008 etc).

For those not familiar with the way that the Windows kernel "works", here's a minimal introduction:

The Windows kernel has two important memory pools, both critical to the correct operation of the kernel. One is the paged pool, able to be swapped to disk as required. The other is the non-paged pool, permanently in RAM. If either of these pools is exhausted, stability will be impacted.

This is why the UNIX world embraces redirection into user space for most high-level tasks. Virus scanning, remote access, graphical interfaces, intrusion detection, complex firewall rules and logging (to name a few that come to mind) are all handled mainly in user space, where crashes and incorrect behaviour are handled with a process kill and respawn.

Windows handles things in a, how shall we say, different way. Drivers for the vast majority of remotely low-level tasks do complex work inside kernel space. There are multiple cases from Network Associates, Symantec, Computer Associates and Roxio of causing massive stability problems due to their drivers. My employer has various requirements mandating the use of various packages from vendors that install an assortment of drivers that do non-trivial things in kernel space. In every case, these drivers have resulted in major stability problems.

This gives rise to two very important questions:

How hard can it be to do a decent amount of testing to find these problems, given they require me to reboot my servers every week to "work around" them?
Why aren't these drivers a boring, stone-simple stub that pass the required data to a user space process that won't bring down the server in question due to a simple development error?

This week, I was faced with an agonising decision. A server running Windows 2003 Server (Standard Edition, 32 bit) that normally shows a Paged Pool usage of 30MB - 40MB was showing usage of 320MB.

After promptly panicing, I went looking for the cause. This has happened before, and I'd maximised the size of both pools. The idea wasn't to prevent the problem, but rather to at least get the chance to both see the problem and diagnose it. We've ensured that Pool Tagging was enabled on all of the Windows machines we take care of and that Poolmon was readily accessible.

As soon as I saw the problem I fired up Poolmon and saw that the tag for the biggest user of the pool (280 MB of usage) was SevI. A quick check showed this was Symantec's SymEvent driver. And this is where my frustration began:

The SymEvent driver isn't advertised as being included with or used by pcAnywhere. It's mainly associated with Norton AntiVirus and Symantec AntiVirus
Executing LiveUpdate against pcAnywhere doesn't update the driver
The latest version of pcAnywhere no longer uses the SymEvent driver
There isn't a program available that will clean up the paged pool usage to allow me to forgo reboot of a critical production server

As much as I disagree with Microsoft's business practices, they actually make very good tools for verifying drivers (PreFAST is actually quite fantastic). You could also use Purify and Valgrind (yes, it runs on Windows). Symantec are also a compiler developer.

Because of some programmer's inability to understand that you should free what you malloc, I was forced to reboot a critical server before the "acceptable" hours of 3am - 6am. I applied the patch, and rebooted to a situation where I am expected to believe the problem is solved.

There are several worrying morals to this awful story:

Windows is too complicated. As my boss does say (he's not a developer or sysadmin), you should always seek to make things more simple
Many eyes make all bugs shallow. Open source software is held up high by geeks for a reason.
Don't use pcAnywhere. There is no possible justification for using it.
The companies expecting us to trust them with our host-based security systems expect our uptime in return without consequence (NAI, Symantec and CA are all guilty)
We (those in the industry) are cynical because we're given valid reasons to be that way

So to those responsible for my pain, those at the biggest security companies in the world, thanks for bringing down our industry in the eyes of the outsiders. You owe we in the trenches an apology, and it's way overdue.

Basics - 1

2008-03-17T00:24:00.003+11:00

The basic things tend to be what trip up complicated applications and systems. For instance, just the other day one of my colleagues asked me what the last packet sent to close a TCP socket connection was. He was working with a package that seemingly wasn't playing by the normal rules.

"FIN" was my reply - and I was right. That's not the point of the story though.

The developer working on this project is extremely good at what he does. The fact that he found this problem quickly and solved it quickly highlights two important aspects of the way he does things:

1. He looked for a cause, not tried a solution without basis
2. He asked for clarification to speed up the fix

The number of cases that exist where the words and phrases "maybe", "probably" and "what else could it be" appear in response to a problem without having a proper look is worrying.

The most important thing to remember when it comes to solving problems is that you should call on the experience and insight of others. Simple really.

Apparently I have cancer. Thanks Internet!

2008-03-16T23:29:00.002+11:00

Yes, according to the Internet, I have cancer:

http://www.healthline.com/symptomsearch?addterm=back+pain&picklist=

Of course, it should be painfully obvious that this is just not even close to being helpful but really - listing cancer, cardiovascular disease and many other things must either be someone's idea of a joke or protectionism gone mad.

Observe that this isn't BMEzine, Rotten, Portal Of Evil or any other shock site but could quite easily have a much more profound effect than those sites on unsuspecting persons (or those silly enough to trust the internet and its contents).

As I always say, browsers should place a modal dialog in front of users upon start up with the following content:

Welcome to the internet. Those responsible are sorry.

If you can't measure it....

2008-03-02T23:57:00.002+11:00

If you can't measure a quantity that you need to examine for your job or current task, then you're focusing in the wrong place.

When it comes to computing, visibility is key to maintainability, serviceability, reliability and success. Visibility means measurement.

This is a concept not lost on those who have come before us, given the multitude of packages and protocols dedicated to this end.

I'm lucky enough to work with people who understand the critical importance of measurability and monitoring. Unfortunately, the industry isn't lucky enough to have had this most basic and fundamental fact become a pervasive truth. Let's take a simple example.

Today, I reinstalled Microsoft's latest attempt at a desktop OS, Vista. My desktop rig has 2 RAID controllers, both of which have WHQL accredited drivers available.

Anyway, the initial leg of the install went perfectly. Upon rebooting and Vista actually launching from the hard disc, I was greeted with the box rebooting at a certain point. While I figured out what the problem was and how to fix it, would an error message of any type or description really be too much to ask? Even a stack trace is more helpful than a reboot.

At a higher level, I have taken care of many networks where monitoring and measurements amounts to testing ping response times and little else. And the examples of this happening were for very big, very valuable networks. Networks of over 100 devices have not had even this level of monitoring.

Thankfully, the excuses of mundane, involved setup routines for monitoring are all but gone these days. Take Zenoss and Hyperic - both excellent packages that provide excellent visibility and monitoring of many different types of software and both easily customisable and extensible.

If you have a system that involves two or more interconnected programs, hosts, networks, switches, routers, firewalls or any other type of device may I suggest you monitor it - for your own sanity if nothing else?

Correlation is best seen when nice uniform measurement is in place and "just happens". The number of problems you find may not change, but the time to resolution and the preemptive options you have will increase and there are no exceptions to this as long as your monitoring is extensive and complete.

More performance = more problems

2008-02-28T00:50:00.003+11:00

Problems are, at their most simple level, manifestations of undesired outcomes.

Of course this is a wide and far-reaching definition, but it serves the purpose for this little diatribe.

Recently, a service was moved from a shared host to a dedicated host. This increased performance considerably and made maintenance of the moved service much easier. All initial tests went perfectly, and the benefit looked to be everything that was expected.

Then, we noticed strange set of characteristics in the application that had been moved. Other systems (not on the shared host) started to display similar characteristics. The most odd of these was an inverse correlation between connection count and thread count through a number of systems. As good as the monitoring that's in place is, accurate data doesn't help with baffling behaviour.

In the end, our added performance was the cause of our problems. We have a few services that are called on by a number of applications. The way one of those services was used had never seen the sort of loads that the reorganisation had resulted in. Our clients loved the additional speed, and they used it.

In the end, making a few minor changes allowed us to retain our performance improvements whilst eliminating the odd behaviour.

A case that is delightfully counter-intuitive, but brilliantly instructive.

We are all connected - A tale of system failure

2008-02-25T23:19:00.003+11:00

Over the last 48 hours, a coworker and I have had to deal with a number of system problems across the many hetrogenous setups we take care of.

This is not unusual in our line of work - we have in-house produced, third-party and software-as-service style systems to take care of. We've seen the symptoms manifest from the known causes in the past as well, but this incident served as a most excellent reminder.

One particular service we take care of does real-time information lookups against a third-party's systems. The interconnection between our gear and theirs is not all that complicated, but does require some understanding and care.

Anyway, the third-party in this case had some network problems that knocked out parts of their systems. The problems were such that no lookups could be done, thus the service on our side wouldn't work.

We have a number of companies we interact with via the same interconnection method, and a number of them share infrastructure on our end. As is so often the case, systems often have follow-on effects to the systems they are connected to. A busy database server can cause problems in an application's web tier for example.

In this case, the interconnectivity used means that, without the due care and understanding, a lack of responsiveness from one lookup source and impact the deliverability of the entire service.

All of this is due to one aspect of the system's overall design - a single choice made many years ago (before myself or any of my workmates were involved - the system was acquired from a competitor).

This single choice was the result of vendor loyalty and narrow thinking. Vendor loyalty, in that using technologies recommended by and/or created by a particular vendor without regard for what other systems and technologies will be involved. Narrow thinking in that a single technology, designed for a specific purpose was considered to be the best solution for the wide array of environments that were part of the project's brief from day one.

With so many systems connected to other systems, there are still worrying recurrances of old, well-worn mistakes. Three small facts should be kept in mind when designing any system that interacts with another:

Networks fail
Bandwidth is finite
Latency changes, constantly

Simple really.

What you know doesn't matter

2008-02-25T01:59:00.004+11:00

I have a number of colleagues that base their job security on one metric and one metric alone.

What they know

Unfortunately when questioned further what they mean becomes clearer and more worrying. It's not that their training, insight or approach is unique and not able to be reproduced. Rather, they are referring to (as a most excellent friend of mine put it) "esoteric knowledge" about their environment. Some choose policy as their instrument of security, some choose belligerence and some choose a combination of these and other foundations to ensure their place in an organisation is secure and indispensable.

To all those who think this way - please take a good look at what you are doing. I can assure you that your security is short term and the impact to yourself and others is at best tolerated. As the original aggregator did recently bring to my attention, those who are good at their job inevitably get away from it, at least in IT. Those who do otherwise are characterised quite simply.

Seemingly counter intuitive perhaps, but entirely normal. A great software developer will not be writing code forever and will instead elevate their peers and eventually those who work for them. A great system administrator will not be writing shell scripts forever and will be asked to build a team to continue their most excellent work. Every DBA seeks to take care of bigger and better resourced data warehouses and can surely not expect to do so alone.

In all industries, we all seek to move into better pay rates, better work conditions and generally better lifestyle.

To actually improve your pay rate against the cost of living, you need to increase your value to whichever organisation you work for (even if, and especially if, you work for yourself) - no amount of hard work will offset the gains made through efficiency and knowledge transfer.

To improve your work conditions, the need to maximise your options for growth and advancement whilst minimising your need to perform mundane and repetitive tasks must surely be paramount.

Improvements in lifestyle are a product of both these things - your family and friends are surely not wanting to see you happy and growing as opposed to satisfied and surviving.

We are all human, and advancement is what separates us from the animals. Not passing on knowledge, empowering those around you and failing to relinquish control serves no one, least of all yourself.

The best move is no move at all

2008-02-25T01:11:00.000+11:00

It's worryingly simple. In all of software development, system administration and database administration, the best move is so often no move at all. In fact, it often applies to life in general.

Yes, it's non-sensical and somewhat circular but is a statement that makes much sense. Thinking about it though:

As a software developer, it is extremely unlikely that you'll come across a problem that no one else has had to solve and solve repeatedly. The first rule of writing great code is not to - if libraries exist, use them!
As a sysadmin you are most probably administering systems that plenty of other people take care of, satisfying similar requirements to those that many of your contemporaries are burdened with. The chances that none of them have posted or blogged their solutions and thoughts are quite low.
As a DBA, unless you have an environment the rest of us envy, then you'll have an environment the rest of us have relevant experience on - please use it.

It was Newton who said "If I have seen further it is by standing on the shoulders of giants" - the man was stupidly intelligent and listening to him makes an awful lot of sense.

If you can find a library, a set of scripts, a whitepaper or a blog entry/post that describes a suitable solution, you'll benefit. One of two things shall happen:

Your problem will be solved, and you can do something else
A possible approach will have been explored and excluded, narrowing your search

Additionally, either outcome will hopefully further contribute to the knowledge at hand of yourself and those who do as you do. If you find a suitable "off-the-net" solution, the mere mention of the specifics of your environment as a comment or reply will extend the suitability of the solution for everyone else. If a solution you find does not fit, mentioning why will either result in the information you found being updated and adapted or people saving time through your contributed analysis.

Simple really - but as the great one did say, it's much harder to be simple.