Business analytics model risk (part 4 of 5): categorizing model risk

Business analytics model risk (part 4 of 5):  categorizing model risk

Following from article 3 of 5 on Business Analytics Model Risk

Link to introductory header article (0 of 5)

Model risk cannot easily be spoken of as a singular entity.  The topic is complicated in that there are multiple ways of categorizing model risk.  As well, schemes for categorizing model risk are rarely mutually exclusive.  While we can easily categorize different types and aspects of model risk, it is difficult to propose exclusive categories.  Aspects of model risk tend to easily overlap, co-occur, or co-vary.

At the outset of this series, a categorization of contributing factors to model complexity was proposed.  Complexity is one type of categorization scheme for a particular type of model risk.  Below two more are offered which are likely more useful in segmenting and thus controlling model risk:  I) model core attributes, and II) modeling process steps.

By proposing sub-categorized aspects of business analytics model risk (which may overlap and interact), we can gain an understanding of particular areas where more attention is needed in the process of designing, validating, and implementing decision models.  The utility relates to improving the practice of sound model implementation, to controlling for model risk via preventative organizational processes and measures.

I.  MODEL CORE ATTRIBUTES

Business analytic model risk

1)  Technical:  reflecting the formal implementation of the business question, set of choices concerning the dataset (i.e. scope, type, size),  methodology to obtain insight (i.e. descriptive statistics, linear, nonlinear, mixed), technologies employed (i.e. spreadsheet, software mediated, custom), and particular approach to validation (i.e. testing and review approaches).

2)  Functional:  particular functional area(s) of business over which model is proposing to guide decision making.  Mediates between phenomenon and a target area of business activity (see Figure 1 from previous article).  Functional areas (i.e. finance, operations, sales / marketing, customer service, HR) each have their own technical-practitioner factors which influence model design.  Challenges can occur in translating or simplifying domain knowledge, particularly between functional experts and technical experts.  As well, models are increasingly complex, spanning several functional domains (i.e. a strategic planning model which uses financial risk analysis to value and evaluate growing untapped customer markets for the purpose of operations capacity planning).

3)  Structural:  composite ‘componentisation’ and process-orientation of aggregate model.  Aggregate models raise issues of non-linearity, reproducibility, maintenance, and comprehensibility.

4)  Organizational:  in terms of building rigor into socio-organizational processes to reduce opportunities for explicit and implicit errors that can creep into modeling and model-based decision making.  This can be more explicitly categorized as ‘agency’ or stakeholder-related model risk. This concerns tacit and/or explicit influences related to agency conflict in the firm, such as power politics, empire building, and general internal competitive factors.  This encompasses bias within and between organizational roles:  capital holders, managers, technical specialists, methodological specialists, area functional experts, external contracted experts, and customers.  Funding and power influences are in particular both influential and quite subtle, for instance the complicity between managers and experts who are direct reports of the manager (or clients and consultants). An example is when a chartering stakeholder (i.e. manager as client) desires a particular decision outcome and subtly (or overtly) influences the recommendations of the chartered expert(s) (i.e. consultant or internal expert).  Otherwise, communities of tight role-based practice, based on isolation or exclusivity, can also easily slip into availability or confirmation behavioral biases. An example is of a group of engineers focusing on technology risk in a project and completely ignoring a financial risk such as currency exchange.

5)  Behavioral:  inbuilt psychological and group-related behavioral biases (such as overconfidence, anchoring, availability) which can threaten model robustness.  Work of Daniel Kahneman is directly relevant.  Can mislead modeling efforts concerning problem identification, framing, data selection, methodology selection, and results interpretation: http://en.wikipedia.org/wiki/List_of_cognitive_biases#Decision-making.2C_belief.2C_and_behavioral_biases

II.  MODELING PROCESS STEPS

Recognizing that the proposed core attributes are often intertwined in complex ways (i.e. technical choices being influenced by both decision biases and organizational factors), it becomes useful to view model risk in terms of a process of steps from design to implementation.  In this context, we can propose three rough model creation steps (also the three steps in risk management):  1) Design, 2) Validation, and 3) Implementation.  These may occur in a iterative fashion, but they result in a general linear flow that ends with organizational use (implementation and maintenance) to guide decision making (often as encoded into an IT system):

1.       DESIGN:  specification – the risk of a wrong model

• Problems of framing: stipulating the wrong business problem and set of questions, particularly when a danger when leaping right into analysis without generating  a robust understanding of the underlying business problem and questions (i.e. attempting to analyze symptoms instead of causes)
• General inapplicability of model:  faulty application of a metaphorical or superficial representation to a literal representation
• General inaccurate specification: faulty theoretical assumptions built into model, such as asserting causation where there is only correlation demonstrated (susceptible to structural equation modeling (SEM) analysis),
• Organizational scoping and composition
  • Overreliance on vested stakeholders
  • Behavioral biases (i.e. overconfidence, availability, grounding)
  • Agency factors (i.e. empire building motivations, perverse incentives)
• Methodological
  • Faulty assumptions of linearity (where linear relationships are indicated but do not exist)
  • Faulty assumptions of nonlinearity (where linear relationships exist, but are not specified / characterized)
  • Lack of transparency (citations, assumptions, and logic not explicitly annotated to establish credibility)
• Uncertainty
  • Inaccurate probabilistic assumptions (i.e. wrong probability distribution chosen, poor distribution fitting to incomplete historical dataset, poor choice in computational randomness)
  • Improper treatment of volatility uncertainty (i.e. volatility smile)
• Model
  • Simplicity versus complexity (level of granularity versus exhaustiveness)
  • Breadth (improper systemic frame)
  • Overfitting (too closely specifying the model such that future unanticipated phenomenon are not accommodated)
  • Compactness (complexity in model leading to unmanageable models, or models which relate to overly-focused circumstances)
  • Addresses infrequent events (allowing for ‘black swan’ scenarios, subject to scenario and ‘what if’ analysis

2.       VALIDATION:  estimation and specification – risk that model is improperly validated

• Testing
  • Lack of exhaustive testing
  • Improper dataset for testing
  • Design errors creep into testing, leading to improper validation
• Dataset
  • Availability / selection bias (selecting a sample dataset which does not adequately characterize the broader population because the data is easily available, convenient, inexpensive to obtain (if under budget pressure), quick to obtain (if under time pressure) or representative of some cognitive bias or stereotype of the analyst and/or stakeholder)
  • Lack of transparency (inability to document and share test data and related assumptions)
  • Breadth (conceptual)
  • Scope (related to scope of design, implying lack of representativeness in test data)
  • Composition (lack of representativeness)
  • Size (too small, principally)
  • Oversimplification (model is representative, but too selective)
  • Inaccurate summarization (poor or non-standard choices in summary statistics)
• Organizational
  • Cursory validation (stakeholders sign-off too early due to lack of effort, too high costs, overreliance on opinion of experts, overreliance on opinion of vested managerial interests)
  • Corrupt and perverse selection (dataset selected specifically validate a pre-disposition or conviction on the part of experts and/or management)
• Maintenance / Re-validation
  • Failure to iterate / review periodically (in particular when there is no plan stipulated for periodic review and maintenance of the model – no ongoing ownership)

3.       IMPLEMENTATION:  verification – the risk of faulty operationalization

• Technical
  • Calculation errors
  • Programming errors
  • Software errors
  • Data implementation errors
• Operationalization
  • General poor operationalization (errors in design and/or specification creep into use)
  • Overreliance (overextension of model use beyond intended scenarios – the ‘magic oracle’ or ‘black-box’ syndrome)
  • Failure to use effectively (although accurate, system not attached appropriately to organizational decision making processes)
• Conclusions / inference
  • General errors of theory (improper theoretical framing such as implying causation where there is only correlation)
  • Type I Errors (incorrect rejection of a correct null hypothesis, leading to failure to refute an incorrect hypothesis and adoption of significance where there is none)
  • Type II Errors (failure to reject a false null hypothesis, leading to acceptance of significance where none is justified)

* This list is a first attempt and feedback is very welcome:  I will update and annotate this listing based upon suggested additions to and revisions to this list.  The final article will propose categories of approaches for dealing with business analytics model risk.  Email:  webmaster@sark7.com

End of article 4 of 5

Link to next article in series (5 of 5): pending June 2013

Link to introductory / header article (0 of 5)

———————————————————————————–

BLOG POSTS / ARTICLES OF INTEREST

Data Scientist: Bias, Backlash and Brutal Self-Criticism:  http://www.ibmbigdatahub.com/blog/data-scientist-bias-backlash-and-brutal-self-criticism

Careful: your big data analytics may be polluted by data scientist bias:  http://gigaom.com/2013/05/04/careful-your-big-data-analytics-may-be-polluted-by-data-scientist-bias

The hidden biases in big data:  http://blogs.hbr.org/cs/2013/04/the_hidden_biases_in_big_data.html

The dictatorship of data:  http://www.technologyreview.com/news/514591/the-dictatorship-of-data/

Evidence-based versus intuitive decision making:  http://blogs.hbr.org/hbr/mcafee/2010/01/the-future-of-decision-making.html

REFERENCES

Ansoff, H. I., & Hayes, R. L. (1973). Roles of models in corporate decision making. Paper presented at the Sixth IFORS International Conference on Operational Research, Amsterdam, Netherlands.

Balci, O. (1998). Verification, Validation and Testing: Principles, Methodology, Advances, Applications, and Practice. In J. Banks (Ed.), Handbook of Simulation. New York: John Wiley & Sons.

Derman, E. (1996). Model Risk. Quantitative Strategies Research Notes. Goldman Sachs. http://www.ederman.com/new/docs/gs-model_risk.pdf

Hubbard, Douglas W. (2009). The Failure of Risk Management: Why It’s Broken and How to Fix It. John Wiley and Sons: Kindle Edition.

Morini, Massimo (2011). Understanding and Managing Model Risk: A Practical Guide for Quants, Traders and Validators (The Wiley Finance Series). Wiley: Kindle Edition.

Sargent, R. G. (1996). Verifying and Validating Simulation Models. Paper presented at the 1996 Winter Simulation Conference, Piscataway, NJ.

Shannon, R. E. (1975). Systems Simulation: The Art and Science. Englewood Cliffs, NJ: Prentice-Hall.