Business analytics model risk (part 3 of 5): model scoping and complexity

Business analytics model risk (part 3 of 5):  model scoping and complexity

Following from article 2 of 5 on Business Analytics Model Risk

Link to introductory header article (0 of 5)

In a the first article in this set on business analytics model validation (https://sctr7.com/2013/03/14/model-risk-when-is-a-business-analytics-model-validated/), it was proposed, in summary, that:

1) business decision models are becoming increasingly complex due to the co-emergence of technology, globalization, communication, and competition factors;

2) growing complexity in business decision models is increasing difficulties associated with model validation, particularly in terms of coordinating consensus and orchestrating clear communication;

3) growing decision model complexity leads to a ‘squeeze’ in which greater numbers of stakeholders are involved, but fewer individuals are able to conceptually grasp the entire model;

4) exhaustive business model validation is not, in any case, methodologically possible, validation being essentially a process of building organizational consensus that the model is ‘good enough’; and

4) due to these factors, therefore the general need for better process-focused structure in processes related to model design, review/validation, and implementation.

complexity

Citing Derman, who speaks in terms of financial model risk, but whose meaning we can apply to broader models: “a model is always an ‘attempted simplification of a reality’, and as such there can be no true or perfectly realistic model. But realism and reasonableness, coupled with simplicity, must remain crucial goals of a modeller, and their lack creates model risk.” (Morini, 2011).  Derman’s assertion, together with the above set of proposals and conclusion, imply the need for better structure in organizational processes associated with model design, validation, and implementation.  However, this can only come via a finer-grained understanding of the components of business analytics models themselves.

Since we have established (in article 2) we are concerned about business analytics model risk, beyond exclusive financial industry model risk, it is appropriate more closely define the scope for business analytics models.  The particular models of interest to business analytics (BA) are those which seek to characterize large, complex systems, often aggregating assessments of technical, economic, and behavioral phenomenon.  Such models are of increasing concern to BA practitioners as businesses seek to characterize larger and more complex systems.  Sustainability, global supply chains, outsourced workforces, multi-stakeholder collaboration, and intricate financial engineering are examples of phenomenon driving increasingly complex models.  Such models also typically necessitate validation by broader and more diverse groups of organizational stakeholders.

Concerning the functional specification of business analytics models, we mean the particular functional area of a business where the model proposes to guide decision making.  Figure 1, below, characterizes the trend to embrace increasingly complex sets and combinations of phenomenon in order to address increasingly intertwined business problems:

Interconnections

Figure 1:  Business analytics models increasingly embrace complex inter-systemic dynamics

The scope of BA models is expanding such that they frequently cross functional domains.  Thus, operations planning models typically involve financial factors and sales projections.  Likewise, a financial planning analysis will look at production factors as related to shifting demand

In highly focused implementations, models may be quite self-contained and discrete, applying to a particular domain.  For example, in financial derivatives trading there may be a particular set of algorithms set-up to assess and react to specific boundary conditions in particular markets.  Such models can be tested within their domain against historical data to achieve some understanding of pragmatic retrospective reliability.  Still, as has been the case in recent trading debacles, there are always circumstances in the market where hidden assumptions outside the model invert and cause monolithic misjudgment (i.e. a market collapse causing massive correlation between instruments as liquidity dissolves and market panic set in).  Even discrete financial trading models sit on top of more complex macroeconomic systems, which connect to political and demographic systems via obscure and complex dependencies.  In this sense, self-contained and discrete models can be useful, but are wishful in disregarding larger systemic connections which can cause ‘black swan’ events.

BA models, although they may incorporate a broad range of phenomenon, typically aim to reduce measurement to financial terms. Net Present Value (NPV) assessment is a typical standard for strategic business analytics modeling, as it allows like-to-like comparisons across projects.  However, to the degree BA models frequently attempt to forecast and predict, they may integrate a broad variety of phenomenon into a central NPV model.  For instance, an NPV assessment of a power plant project must countenance highly volatile, highly integrated long-term factors, such as:  perturbations in the price of coal, production output ranges, operational and maintenance overhead, market competition, electricity demand, political factors (i.e. tax policy changes, regulatory impositions, etc.), and the shifting of broad macroeconomic factors (i.e. interest rates, inflation, market disruption).

These types of broad, aggregative models are known as techno-economic models, in that they attempt to undertake broad, unified financial assessments incorporating highly uncertain economic and technical phenomenon.  Increasingly these models are being extended into another dimension: behavioral.  This is the case, for instance, when consumer behavior, market size in relation to marketing efforts, and the effects of competition are assessed.  As well, increasingly the behavioral aspects of financial markets are a topic of interest in complex models.  In the wake of the financial crisis, an assault on the traditional assumption of the ‘efficient market’ has given way to attempts to assess behavioral dynamics in markets.  Such models can be called techno-economic-behavioral.  Significantly, although it may be possible to validate discrete components of such complex, multi-component models, validating all sub-components does not imply a validated aggregate model (Balci, 1998).

Such composite models introduce model complexity, a unique risk factor in itself as a composite of linear systems quickly becomes nonlinear (discrete and relatively predictable systems become highly uncertain).  Sources and aspects contributing to model complexity is thus worthy of an attempt at categorical treatment.

SOURCES OF MODEL COMPLEXITY (NON-EXCLUSIVE / OFTEN OVERLAPPING)

1.  Stakeholder Complexity:  as models become more complex, they often encompass broader swaths of functional and systemic parts of the business.  This naturally leads to broader groups of stakeholders being involved.  Stakeholder involvement can originate from combinations of roles and interests:  manager / owners of affected resources, vested decision participants, area experts (i.e. engineers, market specialists, technical experts), targets of analysis (i.e. outsourced labor, 3^rd party companies, interested customers), or data providers (i.e. owners or providers of data).  To ensure robust models, some attempt must be made to map and involve interested parties in the process of model design, testing, and implementation.

2. Probabilistic Complexity: as more complex models are generated, a range of variables with associated uncertainty typically aggregate / are centralized in models. For convenience, here we use probabilistic complexity to encompass both raw uncertainty as well as probabilistic factors (although these are typically distinct and are treated separately in predictive analytical models).  The aggregation of multiple uncertain variables in models leads to nonlinear bevavior in predictive models.  Care must be take in terms of segmenting uncertain / probabilistic variables (i.e. distinguishing a raw unknown such as chance of a legal suit from probabilistic factors such as price uncertainty based on historical analysis).  Also, when aggregated, care must be taken in terms of the relative interaction of uncertain variables in the aggregate model.  Sensitivity analysis can help to rationalize and understand the aggregate behavior of multiple uncertain variables (i.e. tornado charts, ranking based on simulation).  Often a particular uncertain variable will subsume or dominate.  For example, currency exchange rate uncertainty may completely overwhelm other variables in a model.  Once tracked and accommodated (i.e. currency hedging plan put in place), a particular stochastic variable can be demoted in the model and other factors can be dealt with (i.e. insured, hedged, offset, retained, etc.).  The like-to-like treatment of multiple uncertain variables in an aggregate model should be dealt with in a methodical fashion which both examines the uncertain variable in isolation and then treats it as a relative component in a composite model.

3. Inter-systemic Complexity:  as models become more complex, they typically involve the interconnection of multiple systems.  This factor overlaps with the topic of probabilistic complexity, but can be distinct in terms of relatively simple (linear) sub-models being linked which creates aggregate non-linear dynamics.  For example, a complex manufacturing planning model may involve a sub-model examining the changing price of a commodity on the open market, changing dynamics in customer demand, plant production capacity, capital planning, personnel planning, and currency and interest rate fluctuations.  By aggregating a number of discrete sub-models into a master model, discrete elements may soon generate an aggregate an inherently nonlinear model in which: 1) there is no single optimal solution (i.e. there are only ‘best guesses’ and ongoing orchestration), and 2) relatively simple perturbations in sub-variables lead to highly chaotic effects in the macro system. A characterization of each sub-system must be refined enough that each factor is properly characterized, but attention must be dedicated to aggregate model ‘manageablity’.  In interlinking and characterizing multiple sub-systems in an aggregate model, it is essential to pay attention to two issues:  1) do not overwhelm the model with a ‘spaghetti’ representation of multiple variables co-varying across an unmanageable model, and 2) care and attention must be dedicated to understanding where the aggregate model may indicate the need for constraints-based management (i.e. Goldratt’s Theory of Constraints http://en.wikipedia.org/wiki/Theory_of_constraints ).  An example of an otherwise simple chain of systems which creates aggregate complex behavior (and can quickly lead to sub-optimal value destruction) is the classical bullwhip or whiplash effect (http://en.wikipedia.org/wiki/Bullwhip_effect):  i.e. creating volume-based price incentives for suppliers who pass on discounts to customers in the context of a sales-bonus driven culture can lead to regular stockouts and shortages which damage supplier confidence and customer satisfaction.  Such systems often are better served by establishing a central constraint, such as a regional warehouse that mandates ‘everyday low prices’ in order to optimize sales and to regularize stock handling.

4.  Functional Complexity:  as per Figure 1 above, functional model complexity involves mixing and matching functional business disciplines into composite models to gain insight.  As each domain contains embedded practices, theories, and assumptions, crossing domains in composite models introduces interdisciplinary complexity.  Spanning expert domains can introduce the risk of integration problems, comprehensiveness, mis-matched underlying assumptions, misunderstandings related to terminology and jargon, and long-term maintenance issues.

5.  Methodological Complexity:  composite methods (i.e. mixing statistical analysis, simulation, and decision trees).  This introduces risk related to the boundary conditions and assumptions of particular methodological techniques.  For instance, mixing linear and nonlinear methods for forecasting treats and views data in very different ways.  Multiple techniques also reduces composite model simplicity, introducing issues related to interpretation as well as long-term maintenance.

6.  Technical Complexity:  composite systems and technologies for handing and processing data to gain insights.  Somewhat self-explanatory, complexity can be introduced in terms of the technical handling of data from cleansing to insight as it moves through multiple platforms and systems.  There may be assumptions and procedures embedded in the data cleansing step which are subsequently not treated properly in subsequent data analysis.  An example concerns selecting samples and the treatment of outliers.  This can be an issue particularly in large enterprise environments when many types of functional IT and analytics experts ‘touch’ the data from inception to results interpretation.  As well, model assumptions can become trapped or wrapped in ‘black boxes’ to the degree there is unclear ownership and a lack of documentation.

This represents an attempt to categorize particular types of model complexity.  Please let me know if you have suggestions for edits/additions and I will update this list as an ongoing reference.

End of article 3 of 5

Link to next article in series: categorizing business analytics model risk (article 4 of 5)

Link to introductory / header article (0 of 5)

REFERENCES

Ansoff, H. I., & Hayes, R. L. (1973). Roles of models in corporate decision making. Paper presented at the Sixth IFORS International Conference on Operational Research, Amsterdam, Netherlands.

Balci, O. (1998). Verification, Validation and Testing: Principles, Methodology, Advances, Applications, and Practice. In J. Banks (Ed.), Handbook of Simulation. New York: John Wiley & Sons.

Derman, E. (1996). Model Risk. Quantitative Strategies Research Notes. Goldman Sachs. http://www.ederman.com/new/docs/gs-model_risk.pdf

Hubbard, Douglas W. (2009). The Failure of Risk Management: Why It’s Broken and How to Fix It. John Wiley and Sons: Kindle Edition.

Morini, Massimo (2011). Understanding and Managing Model Risk: A Practical Guide for Quants, Traders and Validators (The Wiley Finance Series). Wiley: Kindle Edition.

Sargent, R. G. (1996). Verifying and Validating Simulation Models. Paper presented at the 1996 Winter Simulation Conference, Piscataway, NJ.

Shannon, R. E. (1975). Systems Simulation: The Art and Science. Englewood Cliffs, NJ: Prentice-Hall.