The Case for QARP

A curious client recently asked – “Couldn’t financial modeling risk be diminished by introducing a Quality Assurance Review Panel (“QARP”)?”  It was a simple, pragmatic question that left me momentarily perplexed. I then enthusiastically responded “I’ll look into it – it could have legs”.

And so I researched further. A threshold question is the practicable trade-off between commercial efficiency and fastidious accuracy. Although most financial modelers aspire to believe they can always achieve both – the first spoke more of dollars, the second spoke more of risk management.  I quickly realised that “efficiency” challenges could be largely overcome through further training (most financial modelers are self-taught), collaborative ways to share knowledge and hard-earned experience.  Finding pragmatic ways to mitigate risk appeared to be path less considered – particularly providing a second-set-of-eyes to highly-complex, thought-intensive and “high stakes” financial models used extensively throughout professional services, industry, government and education.

Research and business commentary, have justifiably been at pains to point out the sources and prominence of spreadsheet. The European Spreadsheet Risks Interest Group (EuSpRIG), a consortium of academics, researchers and professionals who examine spread sheet risks and develop methods for prevention, has even prepared a list of financial modeling “Horror Stories” so plentiful “that we can’t keep lists of all of them”.  To quote Louise Pryor, an actuary and consultant who specializes in software risk management:

Research shows that at least 80% of spread sheets contain significant errors. That means that of every five spreadsheets, at most one will give the correct results. These errors can make a difference of millions; $24m (Canadian) because of a pasting error, $70m (US) due to a modeling error and $1.2bn down to “an honest mistake”.

The utilisation of a QARP may benefit financial modelers for a number of other reasons:

• 1. Scale: Financial models (or spread sheets) are one of the most powerful, complex and widely used computational tools on earth. Microsoft recently estimated that Excel has 500+ million active users worldwide, with 50+ million using the tool for “heavy duty decision making”.

• 2. High Pressure Environment: Model builders, contributors and reviewers regularly work long hours under significant time pressure, high stress and adhere to incentives systems that are output-measured – conditions not exactly conducive to producing high quality reliable financial models.

• 3.  Complexity. Despite an abundance of industry “specialists”, thousands of DIY educational programs, attendance to top schools and work experience for many leading institutions, the practical reality is that many people in professional services and industry still find financial modeling to be highly intellectually challenging and tricky. Not surprisingly, a recent survey found “complexity/scope for error” ranked as the top chellenge facing financial modelers.

• 4.  Limitations of self-education: Since a majority of financial modelers are self-taught (estimates range from 50-90%) – a more valuable, collaborative way to share knowledge is required.  A transparent way of benchmark financial modeling performance may also be necessary.

• 5. Safeguarding professional decision making. Authorities are unmistakably committed to imposing safety standards when a tangible product, activity or process involving mainstream consumers. They are especially quick to enact legislation or consumer protection when an act or omission may threaten to harm, damage or inconvenience an external party. Notable authorities who have become responsible for championing consumer protection include: Consumer Product Safety Commission (USA); Environmental Protection; Agency or EPA (USA); Fédération Internationale de l’Automobile or FIA (France); Food and Drug Administration or FDA (USA); Health and Safety Executive (UK); National Highway Traffic Safety Administration (USA); National Public Safety Commission (Japan); National Transportation Safety Board or NTSB (USA); Occupational Safety and Health Administration (USA); Royal Society for the Prevention of Accidents (UK); European Aviation Safety Agency or EASA (Europe); National Association of Testing Authorities (Australia) and Hazard Analysis Critical Control Point (Australia). Should we also seek to protect broader categories of consumers – shareholders, welfare agencies, educational institution, small businesses – who daily make multi-million and multi-billion dollar decisions with minimal regulatory oversight.  All decisions have major economic impacts and externalities.  It is not often publicised that spread sheets still support much of the world’s critical corporate infrastructure.

The independent role of QARP may allow them to reach beyond merely improving the technical and commercial quality of financial models.  They can also work to improve the risk management, internal controls, compliance obligations and corporate governance best practice (e.g. Sarbanes-Oxley).  QARP may also be a source for thought-leadership and innovative thinking – including sharing insights into how business intelligence technologies can help take spread sheets into the 21st century.  As one small-business CFO recently told me: “Even one error in a spread sheet will subvert all the controls in all the systems feeding into it”.  The merits of QARP may certainly be worthy of further thought in the years ahead – and just maybe, it could do with a better name?

John Persico, will soon be launching Vumero.com, an online marketplace for the world Excel and Financial Modeling freelancing community.