Estimated reading time: 5 minutes
In today’s world, it is difficult to overestimate the importance of models to businesses, governments and in all of our day-to-day lives. Models have become central to understanding a problem, considering the impact of future decisions and monitoring the progress of plans.
So it’s crucial that we truly understand models and know their limitations.
Recently, poor model management from Public Health England (PHE) resulted in around 16,000 UK cases of Covid-19 being missed from the data collection and test and trace system for at least 8 days. While it is difficult to determine the impact of this error, the implications of not contacting those that could have been infected by this group could have real life and death impacts.
The issue was caused by the use of a model that collated data from a number of source files into an outdated Excel workbook that had a clear limitation on the number of allowable rows of data. PHE didn’t recognise this limitation. When daily case numbers rose above the Excel workbook row limit, the excess cases were not added to the list when collecting information to a central system. A check as simple as comparing the sum of data points in the input files to the number of data points on the output file would have avoided the error.
Model risk for insurers
Looking closer to home, model risk is particularly important for the insurance industry, where complex quantitative models lead to strategic decision making and have significant financial impact on the business. It is essential to use appropriate systems, to recognise the limitations of models and to have checks in place.
The spectre of zero or negative interest rates, as raised recently in a letter from the Prudential Regulation Authority (PRA), is a prime example of where model limitations might be costly. Models which are not set to take non-positive interest rates may fail or give incorrect results. Even if your models can deal appropriately with zero/negative interest rates, unless this is documented and certain, the expense of investigations to check may be sizeable.
Below, we talk a little about model risk and how we at Barnett Waddingham are helping clients to avoid it.
Model risk occurs from the use of models that have fundamental errors against intended use and design objectives and from models being used incorrectly or inappropriately.
Very often, issues remain hidden due to inadequate or flawed testing and validation. Sometimes, a pair of fresh eyes will spot long-standing problems. For example, during a recent model review, we identified incorrect coding, an incorrect data field being used and an out of date expense assumption – all on the same model!
Effective control and management of model risk requires a robust governance framework; i.e. a model risk management framework.
Despite the increasing importance of, and reliance on, models there has been little regulatory guidance on model risk. The Prudential Regulation Authority (PRA) has published** some model risk management principles for stress testing for banks and similar firms, but insurance-related material is limited to supervisory statement SS17/16, which focusses on Solvency II internal models.
But firms can’t limit model risk management to specific models or a subset of models. The framework should cover all relevant models across all business units.
In our opinion, there needs to be central model risk management framework that covers the following areas:
- Model risk governance: involves setting up the firm’s policy that clearly identifies the roles and responsibilities of different stakeholders and defines the standards and controls that must be observed when developing and operating models.
- Model risk appetite: sets out the extent of willingness by the board or management body to accept results from models based on the type of model and compliance with the firm’s model risk governance standards. For example, will the firm accept the use of models that do not fully comply with its governance standards and, if so, in what circumstances?
- Model risk identification: a key part of the framework is to identify, record and maintain a comprehensive model inventory capturing, for all models, risk ratings such as materiality, complexity and validation status etc., alongside more mundane items like model owner, users and functional area.
- Model risk monitoring and reporting: ongoing monitoring of models is necessary to ensure models remain fit for use. Monitoring includes assessing the need for model validation and maintaining appropriate documentation including limitation logs. Management information is required to ensure effective oversight. Reporting should cover, for example, key developments and limitations, the evolving model risk profile and recommendations from owners and external validators.
- Model risk mitigation: actions are needed when a firm’s model risk profile falls outside its risk appetite. Mitigation may include model development, use of expert judgement, or additional validation in light of emerging or changing risks such as pandemic risk.
Although data analysis has already completely revolutionised our everyday retail journeys, in the workplace many key decisions are still based purely on anecdotal evidence or instinct alone. Slowly but surely, the juggernaut is turning and analytics is on the rise in the workplace. Indeed, we are heading towards a new destination where Employer DNA will deliver sustainable, robust and innovative strategies.
"We are heading towards a new destination where Employer DNA will deliver sustainable, robust and innovative strategies."
I started by saying that “DNA is the very material that defines our uniqueness – the very substance that carries the information we need to survive and to thrive.” The same is true for Employer DNA. As you work your way up the rungs of the data analytics ladder, the closer you get to the top, the more you’ll realise that your Employer DNA really does contains the insights you need to both survive and to thrive.
How can Barnett Waddingham help?
Barnett Waddingham is a leading provider of Chief Actuary and other actuarial services for range of insurance clients, which require bespoke model build and validation (including Solvency II internal model validation).
Our team deals with complex actuarial models on a daily basis and we have developed a comprehensive model risk management framework to help companies successfully manage model risk.
We will always tailor our services to meet your specific needs and our experts can assist in a number of ways, including:
- building and testing models
- the review of a single model or suite of models
- helping you to implement a comprehensive model risk management framework.
The most challenging business decisions are often backed by complex models that can cost companies a fortune if a model turns out to be incorrect or inappropriate. By using our cost-effective services, companies can potentially avoid loss making decisions and the associated bad publicity/reputational damage. We believe that a fresh pair of eyes can often find problems.
If you would like to talk about this topic or have any questions in general, please get in touch with your usual Barnett Waddingham contact to find out how we can assist you. Alternatively, please contact me below.
Stay up to date
Get the latest independent commentary and exclusive insights from a range of experts at the forefront of risk, insurance, pensions and investment – tailored to your preference.Subscribe today