Business analytics model risk (part 1 of 5): when is a business analytics model ‘validated’?

Model validation

Business analytics model risk (part 1 of 5): when is a business analytics model ‘validated’?

Link to introductory / header article (0 of 5)

See also – recent post: 12 trends in data analytics

In Jorge Luis Borges’ parable “On Rigor in Science” (“Del rigor en la ciencia”), a lost empire attains such perfection in the art of cartography that a one-for-one scale map of the empire is produced, laid-out across the land.  In time, the creation is understandably deemed useless and subsequent generations ambivalently witness the decay of the map (Borges, 1975).  Borges’ critique is that of a science which seeks to perfectly validate, one-to-one, its experiments such that the efforts and impositions of testing and documentation overwhelm the utility and impact of the proofs.  Effective inquiry must enact efficiencies between the overhead of validation and the utility achieved from experimental models.  This notion sits as the foundation of this proposal regarding business model validation: given the impracticality, and indeed impossibility, of ‘perfect model validation’, what can be considered ‘robust’ from a commercial organizational standpoint? Is it possible to improve upon limitations in current model validation practices, particularly when organizational validation is viewed as an unstructured confidence-building exercise?

Organizations, struggling in an age of information overload and complexity, are seeking methods with which to better forecast, strategize, and prepare for lurking risks and to identify hidden opportunities.  The specific domain of concern here is particularly sensitive to model validation overhead:  business analytics, a practitioner discipline uniting management science techniques and information technology solutions to guide decision making in commercial settings.  A fast growing discipline, business analytics practitioners are increasingly active embedding complex decision models in enterprise information technology systems:  software-driven statistical analysis, large data-set management, business intelligence solutions, and decision support systems (DSS).

The rise of increasingly complex analytics systems are accompanied by intricate decision models.  These models require validation:  testing which confirms that the model approximates the behavior of the system under assessment (Pidd, 2004).  While the techniques and technologies for managing these models and systems are advancing rapidly, the methods for organizationally validating the underlying models themselves remain quite rudimentary and unstructured.  Business analytics model validation processes can be broadly characterized as ad hoc, consensus-based, confidence building exercises (Sargent, 2013).  As such, model validation can be seen as an organizational sensemaking effort: an attempt to deem a model loosely ‘good enough’ in its concordance with target phenomenon.  Beyond this, it is generally recognized that exhaustive validation of an analytical model is not formally achievable, being at best falsifiable (Balci, 1998; Pidd, 2004).

As firms rely on increasingly complex analytical models to drive strategic decisions, yet often lack robust model validation processes, it is proposed that emerging understandings of sensemaking in organizations, particularly socio-structural understandings, can be applied to bolster robustness in the practice of business analytics model validation.  It is proposed to enhance as-is practices for validating business analytics models via the application of Social Network Analysis (SNA).  SNA, as a method which can facilitate socio-structural understandings of organizational consensus building, offers the potential to improve robustness and outcomes in model validation processes.

Business model validation may initially appear to be a somewhat obscure domain.  However, the guiding motivation of this research is that poor model validation, and the ensuing consequences, poor business decisions, are a growing threat to organizations, and thus of central concern to both business practitioners and researchers alike.  The need to manage increasingly complex decisions in business is growing due to the combined forces of globalization, technological advancement, and competition (Castellani & Hafferty, 2010).  Global organizations are beset by expanding complexity bearing accompanying uncertainty (Hubbard, 2009; Taleb, 2007).  Growing global business complexity has resulted in increasingly complex models for strategic decision making.  Decision models in large enterprise settings are attempting to embrace broader stakeholder groups, sets of variables, and frames for uncertainty.  Increasingly complex analytical models, spanning broad timeframes, interconnected markets, and diverse supply chains, are being utilized.  With larger and more complex models comes higher sensitivity to initial inputs and assumptions.

Managers are acutely aware that, despite strengthened risk protocols and structured processes, global organizations are often flying blind in environments of uncertainty and doubt, with little warning concerning lurking crises. Concurrently, a new awareness of the limitations of individual and group decision making are emerging: behavioral economics research has uncovered inbuilt decision biases which commonly short-circuit attempts to improve decision quality, even for the best prepared (Kahneman, 2011; Kahneman & Klein, 2009; Miller & Page, 2007; Nutt, 2002). Faced with increasing complexity and data overload, beset managers are easily tempted to retreat into the comfortable, but often misleading, domain of personal instinct and intuition (Bonabeau, 2003). The myth of the omniscient business leader, guided by superior intuition and instinct, comes under assault in conditions where the amount of interacting variables and available data outstrips the capacity of individuals to effectively correlate and process meaningful and significant results.  In an attempt to strengthen controls, marshal complexity, and bolster against poor judgment, organizations are keenly interested in identifying and adopting model-driven, repeatable analytical decision making processes.

To manage complex decisions, many organizations are embracing advanced business analytics tools and techniques.  A rising elite of forward-thinking firms are embracing concerted systems-driven business analytics decision making as a strategic value driver. Celebrated high-performers such as Wal-Mart, Amazon, Apple, Dell, FedEx, and Zara have ushered in a revolution in strategic supply chain analytics, deriving value from concerted computational forecasting and prediction. Recent research explores the foundations of analytics as a powerful competitive differentiator (Davenport, Harris, & Morison, 2010; Kiron, Shockley, Kruschwitz, Finch, & Haydock, 2011; LaValle, Lesser, Shockley, Hopkins, & Kruschwitz, 2011). Higher levels of organizational analytical capability are correlated with value creation via competitive outperformance (Kiron et al., 2011; LaValle et al., 2011).  In the wake of these dramatic success stories, technical and methodological advancements associated with business analytics have taken root in forward-thinking organizations (Kiron et al., 2011).  As such, there is a great interest in the methods by which organizations can spur the adoption and implementation of analytics-based organizational decision making (Hey, 2010).

The success of business analytics as a value driver has spawned growing practitioner and academic interest. Key research themes frame notions of evidenced-based management, decision-driven organizations, and the concept of organizational analytics maturity levels. The rising trend towards analytics is an emergent phenomenon driven by a desire to not only survive, but to thrive amongst synergistic forces: advancing information and communication technologies, expanding datasets resulting from increased collection and storage capacities, globalization facilitated by growing low-cost communication and transport feasibility, virtual companies woven together by partnerships and outsourced services, expanding supply chain and market complexities, and broadening stakeholder factors associated with sustainability. The emerging status quo for global business represents a complex agglomeration of third and fourth level factors which make traditional intuition-based managerial judgment not only untenable, but increasingly dangerous in terms of provoking unanticipated consequences.

However, with the adoption of advanced business analytics approaches comes increasingly complex models, stakeholder involvement, and decision processes.  Complex business analytics-driven decision processes pose organizational challenges. Business analytics practitioners are faced with challenges related to decision process and model management. Given lack of time, funding, and high overhead, practitioner analytics model design, structure, validation, and valorization processes are often ad hoc and cursory exercises.  These emerging technologies and methods put fresh demands upon organizational leadership: new decision processes must be married to organizational structure and culture to be truly viable.  It has been recognized that increasingly impactful economic decisions are made on the basis of shaky, at times flawed, business decision making models.  Forging a tighter organizational contextual link between analytical models and organizational validation processes would improve the robustness of complex business decision making programs.

Given the organizational challenge of validating and valorizing complex models, an integrated analytical modeling and validation approach linked explicitly to organizational socio-structural context would provide practitioners with a method for streamlining and fostering stronger consensus regarding strategic decisions.

End of article 1 of 5

Link to next article in series: framing the business analytics model risk problem (article 2 of 5)

Link to introductory / header article (0 of 5)

, , , , , , , , , ,

About sctr7

Scott Mongeau is an experienced business analytics consultant. He realizes value by meshing organizational and business insight with state-of-the-art technical expertise. Scott has over 20 years of experience in project-focused analytics functions in a range of industries, including IT, biotech, pharma, materials, insurance, law enforcement, financial services, and start-ups. Scott is a part-time PhD (ABD) at Nyenrode Business University. He holds a Global Executive MBA (OneMBA) and Masters in Financial Management from Erasmus Rotterdam School of Management (RSM). He has a Certificate in Finance from University of California at Berkeley Extension, a MA in Communication from the University of Texas at Austin, and a Graduate Degree (GD) in Applied Information Systems Management from the Royal Melbourne Institute of Technology (RMIT). He holds a BPhil from Miami University of Ohio. Having lived and worked in a number of countries, Scott is a dual American (native) and Dutch citizen. He may be contacted at: webmaster@sark7.com All posts are copyright © 2013 SARK7 All external materials utilized imply no ownership rights and are presented purely for educational purposes.

View all posts by sctr7

Subscribe

Subscribe to our RSS feed and social profiles to receive updates.

Trackbacks/Pingbacks

  1. Business analytics model risk (part 2 of 4): saving the kingdom, one nail at a time… | BAM! Business Analytics Management... - June 5, 2013

    […] Following from article 1 of 5 on Business Analytics Model Risk […]

  2. Business analytics model risk (part 3 of 5): model scoping and complexity | BAM! Business Analytics Management... - June 5, 2013

    […] a the first article in this set on business analytics model validation (http://sctr7.com/2013/03/14/model-risk-when-is-a-business-analytics-model-validated/), it was proposed, in summary, […]

  3. Business analytics model risk (part 0 of 5): framing model risk – the complexity genie and challenge of deciding on decision models | BAM! Business Analytics Management... - June 13, 2013

    […] Validation of business analytics models […]

  4. Twelve Emerging Trends in Data Analytics (part 1 of 4) | BAM! Business Analytics Management… - July 12, 2014

    […] An earlier set of posts goes into detail concerning the challenges associated with businemss analytics model validation:  Business analytics model risk (part 1 of 5): when is a business analytics model ‘validated’? […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 547 other followers

%d bloggers like this: