Hope is not a test strategy and how to get testing right

I’ve seen it too many times. Development projects in testing can end up in a tangled mess when the required thinking is not employed up front. I rarely see people putting in place plans which they believe will fail. People aren’t irresponsible. But you can see that the approach is one that will lead to problems.

The consequences are much greater now

With the speed and distribution power of social media, the reputational risk people are taking with their brands is staggering. You don’t need to look very far for examples where the media have picked up on issues very quickly and it has turned into a PR storm.

Look at this and this, for example. Approximately 675,000 of their transactions were effected and the fallout was cast across all forms of media. It was huge. There are no hiding places so you can expect any failure in processing to hit the news and hit the news fast.

There is hope…but hope is not a test strategy. What do I mean by this?

I talk to lots of people about payment projects all over the world. You can quickly identify the thinking that goes into their test plans and make an estimate on the expected outcome. The mind set can be broken down into two clear areas for their plans. Hopefully Thinking and Trustful Planning

Hopeful thinking

When people are hopeful of delivering a robust test plan the thinking can be categorised by the following approaches. Those responsible for the project…

• Do not expect the worst. If you don’t expect the worst then you can’t adequately test or plan for it.
• Test various components of the infrastructure is isolation rather than emulating the entire end to end process.
• They can’t accurately drill down into the precise location where there is has been a problem. Additionally, they can’t identify how that problem could have a consequence on other parts of the transaction.
• They take comfort in the test capacity they do have.
• They over-egg the number of tests and assume volume provides coverage.

Unknowingly, those responsible are skimming over the required critical analysis but they are able to tick a box which says ‘The Test is Complete.’ In reality they are turning a blind eye, crossing their fingers, praying and clutching at four leaf clovers on the day they go live.

Trust in the approach

The ability to integrate new technology with the old technology can be extremely difficult and the problems are getting more and more complex. People need an approach to testing that they can trust. The mind set people need to adopt, I believe, is as follows:

• Assume the system, or the change, won’t work until I have the evidence to prove it does work. This should be in an environment which can precisely emulate a real life transaction - end to end.
• Can I cover all eventualities? Does the test plan achieve this?
• Am I dependent on external services (APIs)? If so do I have test/emulation coverage?
• Can I run repeatable tests to be confident about the volume of transactions? Is it resilient?
• Is the plan scalable?

New Technology

Payments is a complex business. An end to end card transaction, for example, can pass through more than ten different systems before it can be completed. The implications, as we’ve seen, are huge when it goes wrong. But, let’s be honest, it isn’t a matter of life and death (although I was declined trying to pay for Calpol with a credit card in the States for my 3 year old daughter, who was in pain, and it felt close to it. It was infuriating and I was not happy).

But, let’s look at different form of technology and use the same principles.

Let’s take a trip into the future and you’re in a car showroom several years from now. You like to be the first to take-up new technology and the conversation with the salesman is going well. You’re about to be one of the first owners of a driverless car. You’re thinking of the extra time to relax in the vehicle rather than having to stay focused on the road. You’ll have space to recline and read, sleep, or whatever you like to do when travelling.

Everything is in place but, as you’re a thoughtful kind of person, and want to make sure all occupants in the vehicle are safe, you ask about how the vehicle has been tested. The Salesman confidently tells you…

• All the parts of the car have been tested in a lab in California and each one works perfectly.
• The brakes worked.
• So did the sensor which detects objects (other cars, pedestrians, trees, walls).
• The axles and steering are fit for purpose.
• The batteries, which powers the lights, sensor and all the other electrics in the vehicle are tickety-boo.

The Salesman goes on talking about different parts of the car.

You then ask some more questions….

• Has the vehicle been taken for a test drive in different conditions (urban, sub-urban, country roads)?
• How many hours of testing in different conditions have been completed?
• Have all the electrics been tested when working at different times and how did that effect the battery life?
• Has the wiring system been operational and if some of the wiring system fails what are the back-up measures?
• How does the braking and safety responses work in different conditions (wet, dry, hot, cold).

The answer is that each component of the vehicle was tested in the lab in California and everything works perfectly.

Just how comfortable are you going to be sitting in that car as it accelerates up to 55 kilometres per hour from the forecourt? I’d feel more comfortable with my current car.

Apply the same principles to your payments test plans and are you hopeful or trusting of the process?

Hope is not a test strategy…

My father once said to me…

“If you only have a minute to do something you still have 10 seconds to figure out how to do it better”

As the payments landscape gets more and more complicated I feel it is important to look at testing in a different way. The shareholder value that can be lost by a rushed execution, with insufficient thought into delivery, is huge. It’s why we are challenging organisations with responsibility for delivering innovation to re-think their approach and take a fresh look at how they launch new initiatives.