If I ran the ACCC (The Group of Four).....

For those not in the wonderful country of Australia, we have an odd situation here in terms of IP connectivity.

In most of the world, there are carriers who call themselves "Tier 1" who are deserving of the title - they have world-wide networks, own prodigious amounts of international and domestic capacity and provide services at a reasonable price. These companies achieve such status via success in business - they've become successful by running their companies well and providing products that have captured a market. Even "near Tier 1" carriers have become impressive by the same means.

In Australia, things are a little bit different.

Here we have this delightful monstrosity called The Group of Four. In 1998 the ACCC decided that they would react to various grumbles by other carriers and force four carriers to engage in peering, creating the GoF. for those not interested in Wikipedia's ramblings, the four carriers today are:

• Telstra
• Optus
• AAPT
• Verizon Business

As a result of the ACCC's decision over a decade ago, if any other organisation or individual wishes to provide guaranteed access to or from all areas of the internet that terminates within Australia, they have to (directly or indirectly) source connectivity from one of these companies. The impact of this braindead truth is a bit worrying, in a number of ways:

• Two very large residential ISPs (iiNet and TPG) provide inferior access to their customers to many domestic destinations. Instead of connecting to a GoF member at each of their state Points of Presence, they instead only provide connectivity in Sydney (and in Perth in the case of iiNet, only because they wanted to satisfy WoW addicts). Obviously this choice is a business decision, not a technical one.
• One most excellent carrier (Internode) connects to Optus in almost every state and Verizon in Sydney. Ironically, this can cause inferior access paths to be presented to customers due to the way BGP works. As good as this is for customers in terms of service, it surely drives up Internode's provisioning costs, eating into their profits and resulting in higher prices for customers out of simple business sense.
• Telstra and Optus have significantly lower network costs than any other provider of residential services thanks to their significant access network assets coupled with their completely free access to the  Australian internet. In every case, a host within Australia is a customer, a customer of one of their three settlement-free peers or a customer of one of the two previous groups of company. AAPT's sale of their residential customer base and Verizon's absence from the residential market allows them to concentrate on the more lucrative business space.

All that said, the title of this post is "If I ran the ACCC...". So, without further ado, I would:

• Forcibly dissolve the Group of Four, divulging the details of the agreement brought about in 1998 to the public.
• Require that peering at at least one peering point with an MPLA with an opening peering policy that existed on January 1st, 2011 in a given State or Territory be a requirement for any access to the NBN and associated services in said State or Territory.
• Require that peering at at least one peering point with an MPLA with an opening peering policy that existed on January 1st, 2011 in a given State or Territory be a requirement for any access to radio spectrum auctions.
• Require that peering at at least one peering point with an MPLA with an opening peering policy that existed on January 1st, 2011 in a given State or Territory be a requirement for any awarding of construction contracts for the NBN (yes, I'm looking at you Nextgen Networks).
• Require that utilisation graphs for links to MPLA-governed peering internet exchanges be made public by all participating carriers, to ensure that salespeople do not become the solution to congestion problems.

Before anyone thinks I'm mad, let's look at the arguments against my suggestions:

• But the GoF will have their businesses impacted - No, they won't. Companies that are stupid enough to buy business services directly from any of these companies will continue to be stupid enough to do so. Residential customers of similar inclination will continue to be similarly inclined. Given the treatment carriers receive when attempting to source services from the GoF, it's quite clear that such business arrangements aren't too valuable to the GoF, and my proposed points have precisely zero impact on non-transit products.
• But you're swapping one government-created cartel for another! Equinix, Pipe/TPG and WAIA will be the new Group of Three - If I was to show up and tell David Teoh, Simon Hackett, Michael Malone, Ravi Bhatia and John Linton that they could stop paying Optus, Telstra, Verizon and AAPT for domestic transit, I'm sure they'd provide space in their various PoPs across the country for interconnection free of charge in some sort of non-profit structure. Better yet, we'll just look back in Australia's internet history and ask AARNet nicely to provide a neutral peering option.
• Investment in telecommunications will be stalled thanks to large players having less power - yes, these changes that will result in more competition will disincline the government to spend $43 billion, or not.

Of course, if anyone has any counterpoints to my arguments, I would love to hear them.

EMC, IBM, HP, NetApp and Sun - undeserving monopolists

Storage.

For those of us who use computers, it is the most boring and basic element of involvement.

Adding 200% of the current storage throughput available rarely gives us more than a 10% increase in real-life benefits thanks to applications being more interested in order of operations than scheduling, and operating systems being more interested in scheduling than order of operations.

With the rise and rise (and subsequent fall) of ZFS, the role of the FS to do the right thing in the best way possible without delegating responsibilities to logical volume managers, RAID controllers and/or remote storage targets has been realised (and then discarded).

Thanks to a combination of geeks focusing on issues that are "fun", application developers focusing on issues entirely within their constrained domain and Linux kernel developers ignorantly choosing their targets without any regard for the direct employers and (n + 1) indirect benefactors who make their employment possible (indirect FFS/extX FS inodes - you all know about such things), Linux is in a sad and embarrassing state in this area. DRBD is a great piece of software, but it may very well propagate file system problems as fast as your well-designed (and implemented) network will allow.

Don't take my word for it - there are plenty of people beyond my rambling self who agree.

So, as a direct consequence of this current shortcoming, I am announcing a project (name to be made public in the next week) that will provide architecture and OS independent CDP and near-CDP, with appropriate knobs to allow System Administrators to decide which trade-off they need to allow between performance and availability whilst allowing for implementation of local, near and remote DR in any conceivable combination.

In terms of implementation, the implementation couldn't care less which OS, filesystem or hardware you use. The implementation writes bytes to files as requested - exactly as write() and pwrite() do.

I want to know about writes.... (Linux Kernel Team - EPIC FAIL)

Yet Linux tells you, in no uncertain terms, to fuck off.

Here's my scenario in a current software project:

• Process W attempts to write X bytes to file Y at offset Z
• My filter examines said request and based on some specific criteria (that some overworked, under-financed system administrator is mandated to enforce) allows or denies such request
• I have no desire to diminish the performance of the host running my software more than is absolutely required - this means that implementing a FUSE solution is bad (userspace->kernelspace->userspace->my software->kernelspace->fs verses userspace->kernelspace->my software->kernelspace continue),

Reading the above, it all looks quite routine. For those of you thinking this, you are unfortunately wrong.

It seems that the Linux kernel team have precisely zero regard for commercial reality. I will happily apologise in the most public arena possible if this is incorrect but the supporting evidence is in my favour:

* Windows has provided this functionality in a standard fashion for 15+ years. Please, please, please don't make me explain why this is a terrible indictment. 

* Other UNIXes have offered this functionality in well-defined ways for a decade

* DOS offered such functionality to user-space applications

* Linux Security Modules offered the "hooks" to make this happen, but LSM was let into the kernel without any regard to multiple users of said functionality - the next step was to remove the API. 

* grsecurity, RSBAC, Dazuko, RedirFS, McAfee, CA, AVG and Trend Micro are all at a stupid disadvantage with regard to your indifference to reality - access control to file system assets is a modern requirement, and Linux is alone in providing no standardised API for these applications to provide such in a standardised, efficient and simple-to-implement fashion. The teams responsible for OSSEC, Osiris and Tripwire curse you in a similar fashion.

Before the more versed of you respond:

* syscall hooking - yes, I'll totally interfere in the kernel's operations by inserting code not at all verified by the OS vendor. And while I'm at it, I'll disable any software taking a similar approach, or forgo any decently weighty contract of my own software being in effect and disable SELinux, AppArmor and so on. Grow up.

* FUSE - sure, let's engage in 5 copies of data instead of 2. Learn to count.

* Kernel patching - without taking away from the work of DazukoFS (Dazuko and RedirFS are precisely worthless for this application), DazukoFS doesn't actually work on any Linux distribution that anyone who isn't planning to upgrade in the next 4 months actually uses or is supported by any application that any business actually cares about other than Samba. As good as DazukoFS is, it may as well not exist at the moment (see http://dazuko.dnsalias.org/wiki/index.php/Dazuko-based_Applications if you don't believe me).

So, in summary, it's easier to implement anything that worries about the specific operations on files cleanly on Windows than it is on Linux. One is an OS that I love, promote at every possible opportunity, performs better on every modern hardware platform under the sun, provides more out of the box functionality and the other is Microsoft Windows.

Sorry Linux Kernel Team - EPIC FAIL.

Sydney is not the center of the Internet

Definitely not in Australia, and even more so anywhere else.

Here is a list of ISPs and carriers that have a problem that I shall verbalise on:

• Three*
• Vodafone*
• TPG
• SuperNerd*
• iiNet
• Virgin*
• Nextgen Networks*
• Digital River*

The list could go on, but there is no need.

All of these providers have the following problem - all of their transit is in Sydney. Not their international transit, all of it. The starred entries fail additionally, as they peer poorly, if at all.

Traffic from any of these providers for a Melbourne-terminated connection will be tromboned to Sydney, out to the internet and then back again. In the case of the unstarred entries, they do much better in that they peer at Pipe's Melbourne IX and announce at least part of their network there. This alleviates the problem in many cases, but not all. 

For example, my home ADSL2+ connection is provided by Mr Teoh's elves from TPG. Connecting to gear that uses connectivity from those miserable proles in the Group of Four results in this delightful tromboning I mentioned above. The same is true of iiNet, and iiNet only have a single PoP in Melbourne so they're ironically spending less on their core infrastructure than TPG is.

The point is that the captive nature of the Australian transit market results in such patterns emerging for savvy businesses such as TPG and iiNet - peer everywhere and use transit where needed. As for the other entries, there is another explanation.

They're inept, each and every one of them. Failing to peer in Melbourne, Brisbane or Perth when you have a local Point of Presence AND relying on intercaptial backhaul for all connectivity is just braindead. SuperNerd are extremely guilty of this, as are Nextgen and Virgin. There is no excuse whatsoever to not pay Pipe the comparatively tiny sum of $1000/month for 100Mb/s peering in Melbourne and Brisbane and even less excuse to not connect to WAIX in Perth.

TPG's stance is almost understandable, given their pricing and their incredibly lazy approach to their usage of their SX/PPC-1 bandwidth, but iiNet's stance is a bit more puzzling. They have a large number of customers on-net, they're the 3rd largest ISP by customers in Australia, have managed to get a price on Telstra+Reach transit that makes sense commercially yet still refuse to terminate domestic transit anywhere other than Sydney. Fail!

Tier 1 Servers, or not

At work I always find myself buying stuff to make other stuff possible.

Recently we needed to make a small purchase of about $80k to complete a HA project I've been working on for a while. We decided to ask for quotes from both Dell and an IBM partner for this project, as we've been looking at diversifying our hardware base for a while now.

For some reason that doesn't have any bearing to this story, a company had been harassing one of my coworkers for the opportunity to quote on our hardware needs. They insisted they could save us money and replace our Dell servers with "Tier 1 hardware". This leads to a number of questions.

1. What is Tier 1 hardware?
2. Why do I want it?
3. How much is it worth?

The answers to these questions seem to be:

1. Anything not made by Dell
2. Because some sales person with a target for the month says so
3. More than any sane person should pay

For reasons I am yet to fathom, two quotes were provided by said company. Their quotation for the HP gear was around $90k and their quotation for the IBM gear was around $185k. So much for the saving money part of the bargain.

I find it hard to believe that IBM would actually charge such a premium for their gear. Turns out they don't, as a better specified quotation came in from our preferred IBM partner at around $80k. I've no idea where to start in figuring out how such things happen.

Regardless, I relayed this information back to the all-promising vendor and was told that they'd go back to HP to get a lower quote. They did so, sort of.

In the end I didn't bring the HP quotation to the boss as it was clear that I was getting a bad deal in many ways. The hardware on offer featured higher prices, a lack of integrated management, fewer NICs and a complete lack of redundant and hot-swappable cooling. 

I sent a very to-the-point message to the all-promising-supplier after the fact, quoted verbatim here:

Hi Supplier-Representative-7G,

I've spoken with my MD and he's looked at the quotations we received from Dell, yourself and another IBM partner.

For a number of reasons, such as the HP servers lacking redundant cooling and being the most expensive options by far, we've opted not to engage All-Promising-Supplier as a hardware supplier.

Best regards,

Methodology F

Yet, my phone rings at regular intervals, displaying the number of Supplier-Representative-7G. I do not answer these calls, yet still they come.

The next lesson shall be "Tier 1 connectivity, or not"

NBN - get a clue

I live in Australia. The current Labour government is very limited in their outlook on certain issues.

There is much talk of filtering the web and building out fibre. For fuck's sake - get a clue.

Regarding filtering, anyone with 3 braincells and/or a slight want for dissent can easily circumvent any measure you're proposing to put in place. As most of your constituency are bogans, they'll engage in circumvention not for any reason that serves their own purposes, but because their mate Robbo told them about such possibilities at the pub whilst flooding his gullet with Vaginal Backwash.

There is no mention of IPv6, no mention of resiliency and no statement of application.

What will the average Australian moron do with a 100Mb/s link to the rest of the world? Read poorly composed e-mail from his mate Johnno? Watch the latest YouTube viral in 1080p resolution? Upload pornography featuring his girlfriend/wife to some amateur site or transmit the same to that brilliant bloke ORSM?

The vast majority of Australia's population has no idea how their IP feed is provided. Most of them still refer to such connectivity as "my internet" because they're ignorant of the world they live in and are more comfortable with cars, darts and getting into fights at the local pub that considers Victoria Bitter, Carlton Draft and Toohey's New to be a diverse beer selection and get quite agitated if someone dares remove Today Tonight or A Current Affair from the ludicrously large screen whilst they're engaging in drunken escapism between their drudgery of employment and their unhappy de-facto+spawn home life.

Idiot bogans make up a worrying proportion of our populace. The outcome of my ramblings can be summarised thusly:

• The bogan wants faster. Not because they want or need it, simply because it is available.
• The bogan demands lower prices. It does not matter if their provider of choice is making $1.00 of profit per month, they want a lower price.
• The bogan insists that it is being ripped off and that all decisions of their chosen service provider are designed to disadvantage said bogan. A change to any terms or conditions attached to the bogan's service are clearly a personal attack on the bogan.

It clearly follows that all home IP services in my fair country are designed to snare and placate the bogan. Thus, bogans are the problem.

Greatness

Early in my career, an RMIT professor asked me which I should pursue of two choices - achievement or greatness. My answer was greatness, and he chastised me in a most eloquent way.

How appropriate for a man of age 52 to belittle a child of 19 on issues that he'd spent his career examining and lecturing on. Unfortunately my memory is better than his, and my current thinking allows me to respond to his shallow and morally-determined outlook on the path of the individual.

C, Cobol, VAX, TCP/IP, Cisco, Linux, C++, Java, .Net, Cray, Amdahl, The Art, Google, Microsoft

All of these companies, standards and languages have one simple thing in common - greatness.

A limited group of people was involved in each, and their names are synonymous with their creations and legacy. They are identified for their contribution to the world we live in and the communities they created.

Achievement is a very loose and qualitative term from one point of view. A couple's otherwise unremarkable child has "achieved" by getting the best score in the class on a mathematics test or appearing in the school play. As much as my writing will offend some, such outcomes have precisely zero impact on the limited and wider community in general.

In every single case I have cited, the persons involved aspired to something greater. None of them sought the outcomes of fame, recognition or authority - each had their own particular goals that they achieved.

There are many other groups that have aspired and have not been recognised, but there are examples of this that have given rise to better ideas, more concrete outcomes and further-reaching initiatives. The contributions of such individuals cannot be disregarded.

To the professor who shouted me down almost a decade ago: inferiority - what's it like?

The media is awful

This evening, I watched Fighting - a movie of questionable utility.

The lead actress was, as is always the case in movies, seemingly perfect in every respect physically. Slim, curvy, attractive and well spoken.

Yet, in the movie, she was earning $11.00 an hour, and supporting her daughter and grandmother. At no point was there any indication that she ate particularly well, worked out or slept enough to remain healthy.

How incredibly beneficial it is to look to mediums of entertainment for reflections on society.

Or not.

PacNet are retarded (and so are AINS)

PacNet, the strange and no-more-relevant merger of AsiaNetcom and Pacific Internet (at least, it's of no relevance here in Australia), are retarded.

Looking at Pipe's website, PacNet's Sydney DC is a "Pipe Approved" DC. In theory, this means they peer with Pipe.

Taking an IP in their Sydney DC (say 61.14.142.1) and shoving it in Pipe's "Am I Connected?" tool yields most interesting results.

The Melbourne, Adelaide and Brisbane IX points show the IP as reachable, but NSW does not. Which idiot decided to stop announcing that subnet at Pipe NSW? They even announce it at WAIX.

Take iiNet and Internode - their networks will telegraph the traffic to a peering point they can reach in another city for their Sydney users, and then PacNet will carry the traffic back to Sydney!

PacNet are definitely connected to Pipe in Sydney, as their looking glass gets back onto iiNet's network via Pipe NSW. PacNet peer directly with Internode, but Matthew Moyle-Croft (who answers questions about Internode's network on Whirlpool) has made some very "interesting" statements about PacNet's policies since the merger, and the resultant value of the peering relationship.

Why anyone would use their transit capacity or intercapital backhaul when perfectly good peering options are available at a much lower cost is beyond me. The only other carrier/provider I know of that does this are those inept fools at AINS/SuperNerd. That said, they did manage to get one of their peers to announce one of their subnets today but not route the traffic for some reason I'm yet to be informed of.

Thanks to the internet....

I have learned the following, obviously true things:

• I am obese, almost morbidly so (thanks, National Heart Lung and Blood Institute!)
• Milk is the answer to getting thin, and will also cause large amounts of fat gain (thanks, BodyBuilding.com!)
• My legs hurt when I run because of bad shoes, overtraining, sodium deficiency, too much sodium, or one of 5 different STIs (thanks, Wikipedia and the Better Health Channel!)
• The best way to get from Alaska to Queensland is in a kayak....
• I can empower myself as a man by making a few modifications.
• The pot at the end of the rainbow isn't always filled with gold.
• Now that I know I'm obese, I may as well get some support in getting to can't-leave-the-house-fat.
• I'm apparently Jewish, lactating and keep a bloke locked inside a box in my appartment.

Two words. Epic Fail.