More performance = more problems

Problems are, at their most simple level, manifestations of undesired outcomes.

Of course this is a wide and far-reaching definition, but it serves the purpose for this little diatribe.

Recently, a service was moved from a shared host to a dedicated host. This increased performance considerably and made maintenance of the moved service much easier. All initial tests went perfectly, and the benefit looked to be everything that was expected.

Then, we noticed strange set of characteristics in the application that had been moved. Other systems (not on the shared host) started to display similar characteristics. The most odd of these was an inverse correlation between connection count and thread count through a number of systems. As good as the monitoring that's in place is, accurate data doesn't help with baffling behaviour.

In the end, our added performance was the cause of our problems. We have a few services that are called on by a number of applications. The way one of those services was used had never seen the sort of loads that the reorganisation had resulted in. Our clients loved the additional speed, and they used it.

In the end, making a few minor changes allowed us to retain our performance improvements whilst eliminating the odd behaviour.

A case that is delightfully counter-intuitive, but brilliantly instructive.