19th January 2016

Model Validation: Four Elements of Determining the Accuracy of Your Models

Written by Clayton Mitchell, Senior Manager at Crowe Horwath LLP

 

Model validation is defined within regulatory guidance as “the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives, and business uses.” It also identifies “potential limitations and assumptions, and assesses their possible impact.”

Generally, validation activities are performed by individuals independent of model development or use. Models, therefore, should not be validated by their owners as they can be highly technical, and some institutions may find it difficult to assemble a model risk team that has sufficient functional and technical expertise to carry out independent validation. When faced with this obstacle, institutions often outsource the validation task  to third parties.

The Four Elements

Model validation consists of four crucial elements which should be considered:

thefourelements

Source: Crowe Horwath LLP analysis

1 Conceptual Design

The foundation of any model validation is its conceptual design, which needs documented coverage assessment that supports the model’s ability to meet business and regulatory needs and the unique risks facing a bank.

The design and capabilities of a model can have a profound effect on the overall effectiveness of a bank’s ability to identify and respond to risks. For example, a poorly designed risk assessment model may result in a bank establishing relationships with clients that present a risk that is greater than its risk appetite, thus exposing the bank to regulatory scrutiny and reputation damage.

A validation should independently challenge the underlying conceptual design and ensure that documentation is appropriate to support the model’s logic and the model’s ability to achieve desired regulatory and business outcomes for which it is designed.

2 System Validation

All technology and automated systems implemented to support models have limitations. An effective validation includes: firstly, evaluating the processes used to integrate the model’s conceptual design and functionality into the organisation’s business setting; and, secondly, examining the processes implemented to execute the model’s overall design. Where gaps or limitations are observed, controls should be evaluated to enable the model to function effectively.

3 Data Validation and Quality Assessment

Data errors or irregularities impair results and might lead to an organisation’s failure to identify and respond to risks. Best practise indicates that institutions should apply a risk-based data validation, which enables the reviewer to consider risks unique to the organisation and the model.

To establish a robust framework for data validation, guidance indicates that the accuracy of source data be assessed. This is a vital step because data can be derived from a variety of sources, some of which might lack controls on data integrity, so the data might be incomplete or inaccurate.

4 Process Validation

To verify that a model is operating effectively, it is important to prove that the established processes for the model’s ongoing administration, including governance policies and procedures, support the model’s sustainability. A review of the processes also determines whether the models are producing output that is accurate, managed effectively, and subject to the appropriate controls.

If done effectively, model validation will enable your bank to have every confidence in its various models’ accuracy, as well as aligning  them with the bank’s business and regulatory expectations. By failing to validate models, banks increase the risk of regulatory criticism, fines, and penalties.

The complex and resource-intensive nature of validation makes it necessary to dedicate sufficient resources to it. An independent validation team well versed in data management, technology, and relevant financial products or services – for example, credit, capital management, insurance, or financial crime compliance – is vital for success. Where shortfalls in the validation process are identified, timely remedial actions should be taken to close the gaps.

Contact Information

Clayton Mitchell
+44 020 7842 7159
clayton.mitchell@crowehorwath.com

For an overview of the components of a model risk management programme, see the white paper “Effective Model Risk Management for Financial Institutions: The Six Critical Components.”

Please register or login to add this to your interests.