LINK TO HEADER ARTICLE

LINK TO PREVIOUS ARTICLE (5 of 7)

6.    What particular information is gained from social network analysis and how is it interpreted?

To recap, we have established that adopting analytics culture requires organizational change management.  Beyond analytics technology and expertise, analytics culture depends upon effective organizational decision making practices.  In particular, evidence-based decision making best practices need to be sharpened via a focus on effective processes and supporting organizational architectures. It was asserted that decision effectiveness can be improved via concerted attempts to optimize analytical processes, decision rights, access to information, proper incentives, assessment systems, and communication pathways.

social network

However, it was revealed that the track record for corporate change management initiatives is quite poor.  It was proposed that two factors contribute to ineffective change management: 1) over-emphasizing the organizational chart, and 2) a lack of focus on relational interactions and networks.  Social network analysis (SNA) was proposed as a method for improving understanding of relational factors and engineering organizational network change.  This article goes into detail concerning the information that can be gleaned from SNA.

By characterizing the organization as a set of overlapping networks, beyond a singular management hierarchy or accounting artifact, deep insight into organizational structural dynamics becomes possible.  Collating simple data on communication ties in an organization allows maps of organizational structures to be extrapolated.  SNA provides the capacity to extract detailed maps and network quantitative measures by framing and interpreting data on actors and their relations.

For instance, surveying workers regarding ‘who they go to for advice’ allows one to generate a detailed map concerning exchanges of expertise, some of these exchanges which may be reciprocal, some of which may be one-way.  By aggregating all ‘ties’, a structured set of relations emerges as a network map.  It soon becomes visually and quantitatively apparent who the ‘go to’ people are, as well as those who ‘bridge’ information between subgroups.

Subsequent analysis can be conducted concerning demographics and actor attributes such as gender, seniority, and specialty.  From here, research questions and hypothesis can be framed, such as ‘male workers with higher seniority in this firm tend to have a higher density of incoming connections.’  A change program which aims to increase diversity and involvement amongst younger workers in a particular discipline (i.e. statistical analysis), could then use an understanding of the as-is network to implement a rotation program, incentive system, and/or networking body for a target demographic they wish to work into the organization.

As well, such analysis allows for more sensitivity and care to be taken in reorganizations.  With such insight, it becomes quickly apparent that, for instance, quiet, kindly, and unassuming Lois, the legal secretary, who has been with the organization for 30 years, is a ‘hub’ in conveying key organizational process information throughout the organization, often by passing knowledge through several cohorts.  There are unassuming but crucial ‘Lois’ workers in every organization.

Often, the ‘Lois’ worker is the last person to be picked for the ‘guiding coalition’ in a change management initiative and they may be even at risk for being ‘downsized’ due to age and/or ‘unclear value’ (i.e. lack of visibility at higher management levels).  By second-guessing the biases and interests of the guiding coalition and actually conducting such hands-on quantitative research into the organizational network, change initiatives can avoid the costly mistake of, after giving Lois a cardboard box on Friday, finding on Monday morning that she was running the company from below.

Any type of relational-exchange network can be analyzed within an organization: information flows, respect, reporting lines, cross-functional collaboration, decision making, etc.  The benefit of the SNA approach is that it is an exhaustive method: all agents report the basis of their exchange individually.  The network structure emerges from the aggregation of the network ties. Both formal and informal patterns emerge.  Insight is gained into the existence of cliques, or sub-groups which may be somewhat isolated or disconnected from a process.  Insights from SNA map visualizations often are surprising and lead to rapid insights.  It becomes clear who is not talking to whom, who is hoarding information, who is a bottleneck, etc.

SNA information is quick and relatively painless to gather: a survey is conducted asking each individual for a list of names regarding their collaborations (the object of exchange being targeted). ‘Snowball’ surveys can be conducted in larger organizations, whereby people are asked who their ‘go to’ relations are, those people are surveyed, and the subsequent layers are surveyed in turn.  Pictures of the interconnected network emerge.

Also, data can be gathered from passive sources, especially where electronic records are available (and access is permitted): email exchanges, phone connections, meeting invitations, text messages, etc.  When permitted, electronic records of exchanges can be used to generate rapid, detailed maps concerning who is talking to whom (or not talking too whom) in the organizational network.

Beyond the visual maps, there are more focused quantitative measures which can be extrapolated from the network.  Based on a long sociological research tradition, a number of key SNA measures have been developed which can be determined from network data.

• Centrality: which agents are particularly powerful in terms of being highly connected in the network? Research has shown that power and control resides in being highly connected within a network.  Several particular measures of centrality exist:
  • Degree: this is a simple measure of the number of incoming connections an individual has. For example, a politician or mafia don would likely have high centrality degree scores in their various networks.  This is an indication that many others in the network are connected to the person.  Powerful individuals may have many ‘incoming connections’ (people who consider themselves tied to them), but fewer outgoing connections (their own connections to others which they reciprocate).  A reciprocal connection is considered stronger.
  • Closeness: this is a measure of one’s closeness to all others in the network.  A strong closeness measure indicates that the agent can connect to all members of the network in a minimal number of steps. Thus, this person may not have the greatest number of connections, but is able to quickly reach others in the network through a minimal number of steps. It would be desirable as a stockbroker, for instance, to have such centrality as it would allow them to quickly gather information or to promote a stock via-via in an investor network.
  • Eigen vector: (aka Bonacich’s power) this measures one’s connection to those who are highly connected. Whereas a politician has a large number of incoming connections, a political advisor who councils many politicians would have a high Eigen vector score.  Indeed, research attests that this is a powerful and desirable measure: many types of natural and social benefits accrue to those connected to those who are connected, including physical health and happiness measures. Whereas those with many direct connections bear many risks and pressures, a high Eigen vector agent can be thought of as the person playing a key role ‘behind the scenes’.  Thus, whereas it may be very risky to be a mafia don (i.e. risk of arrest or being ‘knocked-off’), a person who advises many mafia dons accrues great power and reach in a network while presumably avoiding direct risk.
  • Betweeness: this is a measure of the degree to which an agent stands in a central brokerage position across key paths or vectors connecting the network.  A person playing a gatekeeper or intermediary role would have high betweeness. This would be advantageous in a trading network as the agent can benefit from being a key intermediary in transactions across the network.  While being a ‘bridge’ is often a crucial function in highly distributed network, there is a danger in terms of having this being a bottleneck and point of low redundancy.  In a decision making network, a person being the only bridge can be considered a risk in terms of decision quality (the upshot being that more bridges should be put in place to improve the flow of information).

A number of standard quantitative measures of network characteristics can be extrapolated from aggregate connection data:

• Network density: how interconnected is the organization on a particular measure? This is a measure of the number of connections in the network compared to the maximum number of connections possible. A highly dense network may be difficult to change as there are very tight connections amongst the agents. Very sparse networks may indicate a level of inefficiency as agents must go through several steps via-via to pass information.  Further, directions and decisions may be slow or even lost in the effort to span a low-density network.
• Reachability or distance: this indicates the minimum number of steps to traverse the network. Popularized via the concept of ‘six degrees of separation’, this is a measure of the number of average steps it takes to get from one part of the network to another. A company with many management levels and a lack of informal social ties amongst workers may suffer from inefficiency and inaccuracy in conveying information.
• Clustering coefficient: (aka cliques) this indicates the degree to which two agents connected to a third agent are connected to each other.  High clustering indicates that there are tightly bound cliques of interconnected individuals in the network.
• Cohesion: this is a measure of the tightness with which agents are interconnected. A cohesion measure indicates the ease with which a group can be broken-up by removing members (otherwise of interest in change programs when disconnecting a group of change resistors is desired).
• Cores: how many significant sub-networks exist within the network?  This measure can indicate when there are tight tribes or competing / non-cooperating cliques.
• Largest core:  what sub-group is the largest and most powerful in the organization?  This is often useful in determining where core power resides in the organization. It may be, for instance, that the marketing group network is strong and large, indicating the marketing function guides the rest of the organization.
• Sub-structures: similarly there may be particular types of sub structures within the network – isolates (loners), pendants (groups thinly connected to the organization), or bridges (gatekeepers or mediators between two sub-groups, as discussed earlier).
• Structural holes: this indicates a lack of connection between parts of a network.  This may be due to competitiveness or silos, but the end result is a lack of exchange between parts of the network to the degree that there is potential weakness and waste in the network (when high connectionism is advantageous or otherwise desirable for efficiency or accurate information passage).
• Co-membership:  what is the level of co-membership in multiple sub-groups within the network? Low co-membership may indicate many isolated or ‘siloed’ subgroups.
• Connections: what characterizes the nature of the dyadic ties within the network?
  • Tie strength: this is an aggregate measure of intensity, reciprocity, and intimacy in connections between agents. A manager that has reporting authority but also respect and admiration would have high tie strength to direct reports.
  • Homophily: this is the degree to which agents display a preference for connecting with others based on particular attributes such as gender, race, age, job role, status, values or any other particular shared attribute of importance.
  • Multiplexity: this indicates the aggregate number of connections between agents based on different criteria.  For instance, two agents may be friends and work collaborators, indicating a strong relationship.
  • Mutuality/Reciprocity: this indicates if two agents reciprocate a tie. For instance, an agent may consider another a friend, but the other may not share that assessment.
  • Propinquity: this indicates a tendency for actors to have more ties with geographically proximate members

By clarifying the types of quantitative measures that emerge from a SNA program, it becomes clear how a change program can benefit.  The variables which emerge from SNA each have implications concerning particular goals for a change initiative.  For a change initiative focused on adopting analytics culture, those measures which seek to strengthen decision making effectiveness would be of particular interest.

For instance, an organization could be measured in terms of its financial planning and analysis (FP&A) network.  What would emerge would be a map and quantitative measures representing the degree to which the FP&A process is cleanly connected (in terms of agent communication).  It would also become apparent where certain cliques or sub-groups consolidated.  It might become apparent, for instance, that there was a lack of connection between functional business specialists, corporate finance planners, and IT people managing the FP&A systems.  Such an organization could benefit by putting in place FP&A functional sub-teams which would encourage tighter network relations between the various FP&A decision actors.

Examples abound and are constrained only by the goals and scope of corporate change objectives.  In the next and final article, we will consider more specifically how the above measures can be used to plan and adopt analytics cultural change.

END OF PART 6 OF 7 IN A SERIES ON ADOPTING ANALYTICS CULTURE

Link to previous article in series: 5. How can change management be improved via analytics?

LINK TO HEADER ARTICLE

REFERENCES

Burnes, B., & James, H. (1995). Culture, cognitive dissonance and the management of change. International Journal of Operations & Production Management. Vol 15, No 8, 1995.

Burton, R. M., Obel, B., & DeSanctis, G. (2011). Organizational Design: A Step-by-Step Approach (Second ed.): Cambridge University Press.

Cronin, B. (2011). A window on emergent European social network analysis. Procedia Social and Behavioral Sciences, 10, 4.

Cross, R., Liedtka, J., & Weiss, L. (2005). A practical guide to social networks. Harvard Business Review, 83(3), 8.

Cross, R., & Parker, A. (2004). The Hidden Power of Social Networks: Understanding How Work Really Gets Done in Organizations. Boston: Harvard Business School Press.

Huisman, M. (2012). Software for social network analysis  Retrieved August 6, 2012, 2012, from http://www.gmw.rug.nl/~huisman/sna/software.html

Huisman, M., & van Duijn, M. A. J. (2005). Software for Social Network Analysis. In P. J. Carrington, J. Scott & S. Wasserman (Eds.), Models and Methods in Social Network Analysis (pp. 270 – 316). Cambridge: Cambridge University Press.

Kameda, T., Ohtsubo, Y., & Takezawa, M. (1997). Centrality in Sociocognitive Networks and Social Influence: An Illustration in a Group Decision-Making Context. Journal of Personality and Social Psychology, 73(2), 14.

Kilduff, M., & Tsai, W. (2003). Social Networks and Organizations. London: SAGE Publications Ltd.

Knoke, D., Yang, S. (2008). Social Network Analysis. London: SAGE Publications, Inc.

Krebs, V. (2012). Software for Social Network Analysis & Organizational Network Analysis  Retrieved August 23, 2012, 2012, from http://orgnet.com/inflow3.html

Popov, V. (2003). Social Network Analysis in Decision Making: A Literature Review (W. Time, Trans.): PSIRU University of Greenwich.

Prell, C. (2012). Social Network Analysis:  History, Theory & Methodology. London: SAGE Publications Inc.

Prietula, M. J., Carley, K. M., & Gasser, L. (1998). Simulating Organizations. Cambridge, Massachusetts: MIT Press.

Scott, J. (1991). Social Network Analysis. London: Sage.

Seely Brown, J., & Duguid, P. (2000). The Social Life of Information. Boston: Harvard Business School Press.

Tsvetovat, M., & Kouznetsov, A. (2011). Social Network Analysis for Startups: Finding connections on the social web. Cambridge: O’Reilly.

Tushman, M. L., & Fombrun, C. (1979). Social Network Analysis for Organizations. Academy of Management Review, 4(4), 12.

Introduction to a series of five articles on model risk

Here we introduce a series of five articles seeking to frame, define, and categorize business analytics model risk.  The intention is to propose processes and practices for strengthening organizational decision model risk mitigation. The series of five articles treat the following structured topics in sequence:

1. Validation of business analytics models
2. Framing the business analytics model risk problem
3. Business analytics model scoping and complexity
4. Categorizing business analytics model risk
5. Practical business analytics model risk mitigation (pending)

Model Risk

The topic of model risk has rapidly come to the fore as a central concern for large organizations.  Growing complexity in business decision making and an increasing reliance on IT-based decision systems, many of which become ‘black boxes’, has raised the stakes concerning model risk.  This topic has been of particular concern in the finance and banking industries as poor models have been centrally identified as a factor in the U.S. Mortgage Crisis and subsequent Global Financial Crisis. The still unwinding Global Financial Crisis has graphically demonstrated the serious repercussions of ‘bad’ (i.e. incomplete, faulty, or misleading) business decision making models.

As broader industries and organizations, beyond banking and finance, are rapidly adopting complex model-based decision making methods, we are concerned with model risk more generally.  In particular, the growth of ‘business analytics’ and ‘Big Data’ as structured approaches to complex business decision making has raised the stakes for improving decision model quality. Complex decision models often become ‘baked into’ systems, whereby a subsequent overreliance can cause spiraling errors.  ‘Bad’ models are quickly ‘hidden’ or subsumed inside complex systems and procedures in modern large enterprise.

Model risk is here specified as ‘business analytics (BA) model risk’ to distinguish it from financial model risk (market and economic decision and risk models specific to and oriented towards finance and banking industry applications), otherwise the dominant current discourse.  This recognizes that much of the literature output is focused on model risk for the finance industry, but that the scope of the model risk problem is larger and broader (across all industries) and thus deserves a more general discussion and treatment.

Thus, when speaking of model risk, we are referring to organizational decision making in large, complex organizations generally.  Although outside a particular industry, organizational decision models often do come down to financial risk (being the near-universal measure for organizational health and performance).  Also, although decision model implementation may be purely organizational, that is, not associated with IT systems specifically, we are more particularly concerned with decision models as encoded into IT systems:  business intelligence (BI), decision support systems (DSS), manufacturing control systems, predictive machine learning, etc.

In particular we are concerned here with highly complex ‘analytics’ decision models which become encoded in IT systems (algorithmically or otherwise in terms of automated computational data processing and procedures). This recognizes that large, complex organizational decision making is increasingly automated by IT systems which encode and embed decision models.  The term ‘black box’ refers to the tendency for such systems to trap and hide potentially risky assumptions with models.

Organizational decision making is a topic which is difficult to discretize.  There are many modes and methods for decision making in large organizations.  In particular, some champion the role of intuition versus process-focused decision making.  Kanheman and Klein have addressed this topic by specifying conditions where intuition-based decision making can be useful in their article ‘Conditions for intuitive expertise: a failure to disagree’.  They stipulate that “evaluating the likely quality of an intuitive judgment requires an assessment of the predictability of the environment in which the judgment is made and of the individual’s opportunity to learn the regularities of that environment. Subjective experience is not a reliable indicator of judgment accuracy.”

Kahneman and Klein assert that intuition is valuable in very specific venues: environments where experience trumps available data.  A linked implication is that such venues are rapidly disappearing: the growing availability of data combined with swelling business complexity creates environments where intuition is a poor alternative to structured data-focused insight.  Growing business complexity in particular increasingly reduces the type of venues where intuition is a preferable decision modality.

Multi-venue complexity is increasingly the status quo for large institutions.  Business complexity, among others, entails combination and permutations of:

• interlinked and extended supply chains (i.e. component ingredients and commodities sourced from multiple 3^rd parties and providers);
• interconnected global financial/funding infrastructure (i.e. economic interdependency of debt and capital providers);
• trans-national regulatory venues (i.e. tax and incentive regimes);
• consumer/market complexity (i.e. broadened consumer choice and global market competition);
• labor outsourcing (i.e. offshore task componentization);
• and information overload (i.e. availability of immense datasets).

We arrive thus at a situation, promulgated by globalization and technological development, where it is difficult to ‘put the complexity genie back in the bottle’.  There is a temptation to retreat to intuition, yet intuition itself is increasingly ineffective given the complex of factors which transcend the ability of individuals to make sound decisions.  We must progress in the effort to make better decisions in inherently complex environments, yet the decision methods of the past are no longer adequate to the challenge ahead.

The theme of this series thus becomes: we are faced with increasingly difficult business decisions which can only be attacked with structured decision approaches, particularly those which combine large dataset analysis with computational approaches.  This sentiment has been roughly popularized as ‘Big Data’:  the structured practice of ‘business analytics’ in attacking large, complex datasets.  However, applying structured decision making itself requires decision making via models.  The problem is thus ‘deciding upon decision models’.  The growing challenge for large enterprise is to specify robust methods for designing, validating, and implementing robust ‘analytical’ decision models in order to countenance the ‘complexity genie’.

Undergirding this assessment of BA model risk are two key, and troubling, assertions: 1) models are by nature ‘wrong’, and 2) no model can be comprehensively ‘proven to be right’ (validated).  Quoting George Box, “essentially, all models are wrong, but some are useful”.  In addition to being ‘wrong’ at some level, we can never fully demonstrate model ‘wrongness’, formal validation (i.e. resolute scientific falsification) being methodologically and epistemologically impossible (Balci, 1998; Pidd, 2004).  The implied objective is to determine where models are ‘useful enough’ while understanding and managing their inherent ‘wrongness’ (limitations implied by and inherent to their boundary conditions as willful abstractions).

These are important assertions, but perhaps not immediately intuitive, and thus shall be ‘unwrapped’ carefully in this series.  The resulting main assertions, and central problems, concerning business decision models that will be explored and treated in this series are:

1. All models, being abstractions of reality, are essentially ‘wrong’ at some level;
2. As abstractions, models can never be demonstrated as being ‘scientifically true’ (i.e. fully falsifiable):  it is not possible to fully validate a complex business model;
3. There is a declining business utility to the management of complex models, implying that there is a pressure in the commercial sphere to achieve simple ‘good enough’ models;
4. Designing and validating commercial models is, in the end, an exercise in organizational confidence building (establishing comprehensive model ‘rightness’ or ‘wrongness’ being both practically and methodologically impossible);
5. Commercial enterprise is particularly susceptible to the twin influences of agency factors (i.e. power politics) and behavioral factors (i.e. decision biases which emerge in complex environments, particularly where time pressure and a lack of robust information is resident); and
6. Robustness in designing business decision models (deciding about decision models) ultimately comes down to engineering better organizational practices and processes to ‘weed out’ the inherent tendencies of groups and individuals to sabotage ‘best practices’ in organizational decision making (both overt/explicit and covert/tacit).

The following article begins with a detailed exploration of the impossibility of comprehensive model validation.  This presents a challenge for business analytics practitioners:  how do we establish ‘usefulness’ or general, practical ‘good enough-ness’, which is otherwise the objective of this series.  How do we best decide on our decision models, given that models are ‘wrong’ and cannot be proven comprehensively?  This core challenge specifies the base conditions for accommodation: understanding and admitting the problem fully is the first step towards addressing it in a practical sense.

End of introduction to a series of five articles on model risk

LINK TO NEXT ARTICLE IN SERIES ON MODEL RISK (1 of 5): When is a business analytics model ‘validated’?

REFERENCES

Ansoff, H. I., & Hayes, R. L. (1973). Roles of models in corporate decision making. Paper presented at the Sixth IFORS International Conference on Operational Research, Amsterdam, Netherlands.

Balci, O. (1998). Verification, Validation and Testing: Principles, Methodology, Advances, Applications, and Practice. In J. Banks (Ed.), Handbook of Simulation. New York: John Wiley & Sons.

Derman, E. (1996). Model Risk. Quantitative Strategies Research Notes. Goldman Sachs. http://www.ederman.com/new/docs/gs-model_risk.pdf

Hubbard, Douglas W. (2009). The Failure of Risk Management: Why It’s Broken and How to Fix It. John Wiley and Sons: Kindle Edition.

Kahneman, D. (2011). Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.

Kahneman, D., & Klein, G. (2009). Conditions for Intuitive Expertise. American Psychologist, 64(6), 11.

Morini, Massimo (2011). Understanding and Managing Model Risk: A Practical Guide for Quants, Traders and Validators (The Wiley Finance Series). Wiley: Kindle Edition.

Pidd, M. (2004). Computer Simulation in Management Science. New Jersey: John Wiley & Sons, Ltd.

Sargent, R. G. (1996). Verifying and Validating Simulation Models. Paper presented at the 1996 Winter Simulation Conference, Piscataway, NJ.

Shannon, R. E. (1975). Systems Simulation: The Art and Science. Englewood Cliffs, NJ: Prentice-Hall.

LINK TO HEADER ARTICLE

LINK TO PREVIOUS ARTICLE (4 of 7)

5.    How can change management be improved via analytics?

We have built-up the case for using organizational change management to adopt analytics culture.  By analytics culture, we mean the set of both formal and informal organizational attributes which emphasize robust decision making. Thus, analytics culture can be considered a byword for evidence-based decision making.  In particular, analytics culture implies organizational processes and practices which emphasize the use of analytics IT systems (BI, statistical analysis packages, and predictive analytics systems and software) and ‘Big Data’ approaches (solutions focused on managing and mining large sets of data to gain business insights).

Organizational network

As we have proposed that decision making is, at base, a factor of organizational processes and behaviors, it becomes natural to frame analytics culture as, beyond technology, a program requiring organizational change.  However, a wrench was thrown into the works by detailing the unreliable track-record of organizational change management. Simply put, organizational change often results in less-than-ideal outcomes.  Thus, there is a risk factor of worsening decision practices by attempting to engineer analytics cultural adoption.

There is hope: it lies in the application of analytics to organizational change processes themselves.  Promising new methods are emerging from sociological and organizational research which can help to make change management less of an uncertain art, and more a structured science and informed technique. In particular, social network analysis (SNA), a technique for the quantitative analysis of social interactions, is a promising method by which change management initiatives can be framed and managed.

This rise in computing power combined with sophisticated analytical tools creates the ability to countenance ‘deep’ socio-structural phenomenon in organizations. SNA is specifically advocated as a methodological approach to quantify and analyze organizational structure for the purposes of managing change programs. Although present in the sociological tradition since the 1930’s, and having proliferated across a number of social science disciplines, SNA is also a trending contemporary organizational research method, bolstered by the increasing visibility of social networks and powerful new computational tools.

The growing popularity of SNA-based organizational research methods is both: a) paradigmatic, bolstered by the emergence of social network media as a powerful and omnipresent cultural zeitgeist (i.e. LinkedIn, Facebook), and b) methodological, supported by the proliferation of increasingly powerful quantitative techniques, themselves facilitated by powerful analytical software tools and steadily growing computational power. Expanded data collection, storage, and handling capabilities have been matched with increasingly powerful software tools for conducting advanced SNA (i.e. UCINET, Sienna, Pajek, specialized R packages).

Conceptually, the key to applying SNA to gain insight into organizations is viewing organizations as networks of agents who are bound together by transactions or exchange-based ties.  Workers are role-based ‘agents’ or ‘nodes’.  These agents interact via transactions, which can be reporting relationships, exchanges of information, sharing of knowledge, co-processing a good or service, submitting requests for assistance, or simply expressed friendship and social affinities. Organizations in this context are network-based decision making mechanisms populated by various interacting agents who also inhabit sub-groups and cliques. Individuals are role-based agents interacting within networks via communication paths to process information into decisions. Workers, as organizational agents, both operate within and compose the network based upon various significant exchanges with other agents.

In the larger context, organizations can be viewed as dynamic information processing ‘organisms’ characterized by a network structure: systems which process information based on unique network structural characteristics and conditions. Agents, tracked as ‘nodes’ via SNA, can be quantified with the full range of attributes of interest to change orchestrators: expertise/experience level, role, affinity towards change, general attitude, and even ‘softer’ measures, such as respect in the organization or ‘buy-in’ on change program.

Likewise, agents-as-nodes can be tracked and linked to other agents via a range of relational ties, exchanges that define one’s participation in an organization.  Examples include communication frequency, respect, membership in cliques, and ties to central processes or systems. Of key concern are both the functional and social / relational ties which define an agent’s participation in the organization.  In the next article we will dig in deeper, specifying the types of quantitative measures that emerge from SNA-based analysis.

END PART 5 OF 7 IN A SERIES ON ADOPTING ANALYTICS CULTURE

LINK TO NEXT ARTICLE IN SERIES (6 of 7) 

LINK TO HEADER ARTICLE

complexity

REFERENCES

Bonabeau, E. (2003). Don’t Trust Your Gut. Harvard Business Review.

Bruggeman, J. (2008). Social Networks: An Introduction. Wiltshire, UK: Rutledge.

Burnes, B., & James, H. (1995). Culture, cognitive dissonance and the management of change. International Journal of Operations & Production Management. Vol 15, No 8, 1995.

Keim, B. (October 2010). Culture evolves slowly, falls apart quickly.  Wired.com. Retrieved October 2010 from http://www.wired.com/wiredscience/2010/10/evolution-of-culture/

Kilduff, M., & Tsai, W. (2003). Social Networks and Organizations. London: SAGE Publications Ltd.

Kiron, D., & Shockley, R. (2011). Creating Business Value with Analytics. MIT Sloan Management Review, 53(1), 10.

Kiron, D., Shockley, R., Kruschwitz, N., Finch, G., & Haydock, M. (2011). Analytics: The Widening Divide. MIT Sloan Management Review (Special Report), 21.

Knoke, D., Yang, S. (2008). Social Network Analysis. London: SAGE Publications, Inc.

Kotter, J. P., & Cohen, D. S. (2002). The Heart of Change. Boston, MA, USA: Harvard Business School Press.

Kotter, J. (January 2007). Leading change: why transformation efforts fail. Harvard Business Review. January 2007.

LaValle, S., Hopkins, M. S., Lesser, E., Shockley, R., & Kruschwitz, N. (2010). Analytics: The New Path to Value. MIT Sloan Management Review, 22.

LaValle, S., Lesser, E., Shockley, R., Hopkins, M. S., & Kruschwitz, N. (2011). Big Data, Analytics and the Path from Insights to Value. MIT Sloan Management Review, 52(2), 13.

LINK TO HEADER ARTICLE

LINK TO PREVIOUS IN SERIES (3 of 7)

4.  Why is the change management track record so poor?

Why are the results of change management initiatives so unpredictable and the outcomes so uncertain?  Earlier in this series, it was proposed that adopting analytics culture inherently requires organizational change, particularly to decision making processes and supporting incentives.  However, evidence suggests worse-than-even odds for success in change management initiatives. In terms of analytics-focused change, the implication is that it is difficult to orchestrate a shift in decision rights, roles, processes, and motivations. In this piece, we focus on the actual mechanics of a change management effort, particularly the role of hidden biases and agency interests which often lead to less-than-optimal outcomes.

downsizing

In the previous article we suggested that a core challenge: many change initiatives fetishize the organizational chart – formal reporting relationships and top-down management structures.  In a Harvard Business Review article entitled ‘The decision driven organization’, it is proposed that change initiatives need to transcend organizational charts.  The article proposes that the focus should be on decision effectiveness via goal-oriented incentives, assessment schemes, access to information, and decision rights. The implication is that the crux of change rests upon being able to shape the structural networks which compose the organization.  In particular, there is a need to affect motivations in terms of social networks and incentives: the imbedded psychological contracts employees have with their organizations.

‘Organizational culture’, the set of attitudes, informal relationships, networks of respect and ‘followership’ (to adopt a term from Barbara Kellerman), determine whether change will be embraced and adopted or resisted, actively or passively.  Passive resistance is the most difficult to detect – it involves ‘disengaged behavior’ and covert blocking that cannot easily be measured or tracked.  However, there is some cause to consider that disengaged behavior stems from a perception on the part of the employee of a lack of organizational justice – a perception that the organization has broken a perceived contract and therefore does not deserve full engagement.

Change leaders largely attempt to motivate change by creating a sense of urgency and by ‘storming’ the culture by putting a coalition of change leaders in the spotlight.  Typically, ‘downsizing’ key dissenters is used to to instill a sense of urgency.  When formal efforts at intercession regarding cultural change fail, change initiatives can quickly devolve into witch hunts where passive resistors and lurking naysayers are routed out and demoted, discredited, or simply walked-out of the organization. This culminates in a codified ritual common to reorganizations, now very familiar to the modern corporate workers: a surprise Friday all-hands meeting leads to a set of sub-meetings in which resistors are called into a scripted ‘career development’ meeting administered by HR specialists.

Packages are offered and career counseling is provided as an ‘invitation to take the next step in one’s career’ (a step which will occur somewhere else, not in the organization). Cardboard boxes are provided from a stack and workers, some dejected, some in tears, collect snow globes and family photos from their desks under the watchful eye of rented security personnel.  IT passwords are revoked, badges are handed in, and targeted resistors and those with nothing to offer the new organization are walked out into the parking lot to begin their now jobless ‘next career adventure’.

I have seen many of these change programs conducted in large organizations and chances are you have as well if you have been involved in corporate work in the past two decades, especially in the U.S.  For those who have experienced a change program, there are givens: 1) it will proceed loosely in a linear, clockwork fashion, as described above (according to the basic Kotter process), 2) despite the structured approach, it is inevitably a tumultuous affair, draining in terms of aggregate organizational productivity and resources, and 3) there will be those who gain, those who lose, and those who must leave.

In the base assessment, people’s jobs are at stake, meaning emotions and drama run high throughout the change process. All that is certain is that some will lose, some will win, and keeping one’s head down is no guarantee that one will emerge from the process unscathed. A range of emotions and behaviors emerge, some contemptible, some heroic.  Viewpoints are expressed and long-led animosities may emerge in dramatic form. When the ‘guiding coalition’ is formed, ambitious middle managers often see an opportunity to increase their power and standing in the organization: posturing and opportunistic behavior emerges. With key stakeholders in the organization knowing radical organizational change is pending, many see opportunities to settle long-held scores and to dispatch enemies or to promote private cliques and cabals.

If we take history as a rough guide, revolutions are tumultuous and destructive affairs.  After an initial heady period of enthusiasm, the revolution classically descends into an entrenched conflict of special interests and vested coalitions.  Often the ‘landed class’ (or vested professional middle class) coalesce into intransigent power interests.  This can be similarly seen in corporate reorganizations:  coalitions of senior managers band together into interest groups and ‘sue for peace’, albeit with the interests of the smaller coalition seeking patronage concessions.  This can have the effect of new middle-management groups initiating a ‘power grab’.

External consultants, often key ‘framers’ and orchestrators of the change process, are guided by a singular metric: client satisfaction.  Thus, consultants, especially to the degree that they lack deep knowledge of the organization’s history and context, are often bound to and guided by the biases and prejudices of senior managers and the subsequent guiding coalition.  There is thus a danger of dispensing the proverbial organizational ‘baby with the bathwater’ to the degree that such efforts ultimately defer to the unquestioned whims of those who fund the reorganization effort.  Little constraints or ‘second guesses’ are put upon those funding the effort.

While none of this is in-of-itself ‘wrong’ (it is what it is and it is, how things are typically done and run), the reality is that the process of ‘unfreezing’, changing, and ‘re-freezing’ does not guarantee that the outcomes from reframing will achieve value creating or even organizationally efficient ends (i.e. core process improvements, evidenced cost reduction, increases in speed, increased customer satisfaction, or improved sales figures). As the Bain research attests, chances are roughly 70% that the initiative will fail outright. All that is certain is that there will be a new organizational structure and that not everyone initially involved will be along for the ride.

Emerging talent will likely have a chance to ascend in the organization via the aforementioned ‘guiding coalition’ (or by brokering/negotiating new roles via the coalition), but the collateral damage can be severe: talented and long-serving employees with embedded knowledge may be inadvertently walked-out, or may walk-out, exasperated by the seeming baseness and often arbitrary process. For employees that are more introverted, there is a risk of being mis-labeled as a resistor or ‘deadweight’.  Some have even charged that there is a predisposition to dispense with older workers with higher salaries and perceived lower energy levels,

In short, great uncertainty afflicts most change initiatives, leading to often arbitrary changes guided by little more that the biases and interests of senior management and the guiding coalition.  Such events are ripe venues for poor decisions to be made, decisions suffused with agency interests (self-serving managerial interests) and/or well-documented cognitive biases (i.e. Dunning-Kruger effect, bandwagon, confirmation, attention, availability, overconfidence, framing, grounding).

For organizations seeking to implement a change program to adopt ‘analytics culture’, to improve ‘analytics maturity’, the risks are even higher: evidence-based management programs propose to tinker with the very gears and cogs of organizational power – access to key information, decision making rights (power), and control over resources (the power to force compliance depending on the framing of analytics).

This then appears to be a sloppy state of affairs, especially when so much organizational value is at risk.  However, as has been suggested in the previous article, there is hope for improved, more ‘surgical methods’.  Analytical methods themselves can be used to conduct such operations with greater rigor.

In particular, it is proposed that social network analysis (SNA) can be used to gain insight into decision and power dynamics in the organization. By ‘mapping’ the networks which compose the organization, targeted diagnostics can reveal where hidden fractures and weaknesses can be addressed. SNA can reveal where there are broken functional processes, decision making chains, and information sharing.  It is to this topic we will turn in the following article.

REFERENCES

Blenko, M. W., Mankins, M. C., & Rogers, P. (June 2010). The decision-driven organization. Harvard Business Review, June 2010, p 54 – 62. Last retrieved May 5th, 2013 from http://hbr.org/2010/06/the-decision-driven-organization

Kotter, J. P., & Cohen, D. S. (2002). The Heart of Change. Boston, MA, USA: Harvard Business School Press.

Kotter, J. (January 2007). Leading change: why transformation efforts fail. Harvard Business Review. January 2007. Last retrieved June 2^nd, 2013 from http://hbr.org/2007/01/leading-change-why-transformation-efforts-fail/ar/

Kotter, J., & Schlesinger, L. (2008). Choosing strategies for change. Harvard Business Review. July – August 2008.

Merchant, K. A., & Stede, W. A. V. d. (2003). Management Control Systems: Performance Measurement, Evaluation and Incentives. London: Prentice Hall.

END PART 4 OF 7 IN A SERIES ON ADOPTING ANALYTICS CULTURE

LINK TO NEXT IN SERIES (5 0f 7) 

LINK TO HEADER ARTICLE

Following from article 3 of 5 on Business Analytics Model Risk

Link to introductory header article (0 of 5)

Model risk cannot easily be spoken of as a singular entity.  The topic is complicated in that there are multiple ways of categorizing model risk.  As well, schemes for categorizing model risk are rarely mutually exclusive.  While we can easily categorize different types and aspects of model risk, it is difficult to propose exclusive categories.  Aspects of model risk tend to easily overlap, co-occur, or co-vary.

At the outset of this series, a categorization of contributing factors to model complexity was proposed.  Complexity is one type of categorization scheme for a particular type of model risk.  Below two more are offered which are likely more useful in segmenting and thus controlling model risk:  I) model core attributes, and II) modeling process steps.

By proposing sub-categorized aspects of business analytics model risk (which may overlap and interact), we can gain an understanding of particular areas where more attention is needed in the process of designing, validating, and implementing decision models.  The utility relates to improving the practice of sound model implementation, to controlling for model risk via preventative organizational processes and measures.

I.  MODEL CORE ATTRIBUTES

Business analytic model risk

1)  Technical:  reflecting the formal implementation of the business question, set of choices concerning the dataset (i.e. scope, type, size),  methodology to obtain insight (i.e. descriptive statistics, linear, nonlinear, mixed), technologies employed (i.e. spreadsheet, software mediated, custom), and particular approach to validation (i.e. testing and review approaches).

2)  Functional:  particular functional area(s) of business over which model is proposing to guide decision making.  Mediates between phenomenon and a target area of business activity (see Figure 1 from previous article).  Functional areas (i.e. finance, operations, sales / marketing, customer service, HR) each have their own technical-practitioner factors which influence model design.  Challenges can occur in translating or simplifying domain knowledge, particularly between functional experts and technical experts.  As well, models are increasingly complex, spanning several functional domains (i.e. a strategic planning model which uses financial risk analysis to value and evaluate growing untapped customer markets for the purpose of operations capacity planning).

3)  Structural:  composite ‘componentisation’ and process-orientation of aggregate model.  Aggregate models raise issues of non-linearity, reproducibility, maintenance, and comprehensibility.

4)  Organizational:  in terms of building rigor into socio-organizational processes to reduce opportunities for explicit and implicit errors that can creep into modeling and model-based decision making.  This can be more explicitly categorized as ‘agency’ or stakeholder-related model risk. This concerns tacit and/or explicit influences related to agency conflict in the firm, such as power politics, empire building, and general internal competitive factors.  This encompasses bias within and between organizational roles:  capital holders, managers, technical specialists, methodological specialists, area functional experts, external contracted experts, and customers.  Funding and power influences are in particular both influential and quite subtle, for instance the complicity between managers and experts who are direct reports of the manager (or clients and consultants). An example is when a chartering stakeholder (i.e. manager as client) desires a particular decision outcome and subtly (or overtly) influences the recommendations of the chartered expert(s) (i.e. consultant or internal expert).  Otherwise, communities of tight role-based practice, based on isolation or exclusivity, can also easily slip into availability or confirmation behavioral biases. An example is of a group of engineers focusing on technology risk in a project and completely ignoring a financial risk such as currency exchange.

5)  Behavioral:  inbuilt psychological and group-related behavioral biases (such as overconfidence, anchoring, availability) which can threaten model robustness.  Work of Daniel Kahneman is directly relevant.  Can mislead modeling efforts concerning problem identification, framing, data selection, methodology selection, and results interpretation: http://en.wikipedia.org/wiki/List_of_cognitive_biases#Decision-making.2C_belief.2C_and_behavioral_biases

II.  MODELING PROCESS STEPS

Recognizing that the proposed core attributes are often intertwined in complex ways (i.e. technical choices being influenced by both decision biases and organizational factors), it becomes useful to view model risk in terms of a process of steps from design to implementation.  In this context, we can propose three rough model creation steps (also the three steps in risk management):  1) Design, 2) Validation, and 3) Implementation.  These may occur in a iterative fashion, but they result in a general linear flow that ends with organizational use (implementation and maintenance) to guide decision making (often as encoded into an IT system):

1.       DESIGN:  specification – the risk of a wrong model

• Problems of framing: stipulating the wrong business problem and set of questions, particularly when a danger when leaping right into analysis without generating  a robust understanding of the underlying business problem and questions (i.e. attempting to analyze symptoms instead of causes)
• General inapplicability of model:  faulty application of a metaphorical or superficial representation to a literal representation
• General inaccurate specification: faulty theoretical assumptions built into model, such as asserting causation where there is only correlation demonstrated (susceptible to structural equation modeling (SEM) analysis),
• Organizational scoping and composition
  • Overreliance on vested stakeholders
  • Behavioral biases (i.e. overconfidence, availability, grounding)
  • Agency factors (i.e. empire building motivations, perverse incentives)
• Methodological
  • Faulty assumptions of linearity (where linear relationships are indicated but do not exist)
  • Faulty assumptions of nonlinearity (where linear relationships exist, but are not specified / characterized)
  • Lack of transparency (citations, assumptions, and logic not explicitly annotated to establish credibility)
• Uncertainty
  • Inaccurate probabilistic assumptions (i.e. wrong probability distribution chosen, poor distribution fitting to incomplete historical dataset, poor choice in computational randomness)
  • Improper treatment of volatility uncertainty (i.e. volatility smile)
• Model
  • Simplicity versus complexity (level of granularity versus exhaustiveness)
  • Breadth (improper systemic frame)
  • Overfitting (too closely specifying the model such that future unanticipated phenomenon are not accommodated)
  • Compactness (complexity in model leading to unmanageable models, or models which relate to overly-focused circumstances)
  • Addresses infrequent events (allowing for ‘black swan’ scenarios, subject to scenario and ‘what if’ analysis

2.       VALIDATION:  estimation and specification – risk that model is improperly validated

• Testing
  • Lack of exhaustive testing
  • Improper dataset for testing
  • Design errors creep into testing, leading to improper validation
• Dataset
  • Availability / selection bias (selecting a sample dataset which does not adequately characterize the broader population because the data is easily available, convenient, inexpensive to obtain (if under budget pressure), quick to obtain (if under time pressure) or representative of some cognitive bias or stereotype of the analyst and/or stakeholder)
  • Lack of transparency (inability to document and share test data and related assumptions)
  • Breadth (conceptual)
  • Scope (related to scope of design, implying lack of representativeness in test data)
  • Composition (lack of representativeness)
  • Size (too small, principally)
  • Oversimplification (model is representative, but too selective)
  • Inaccurate summarization (poor or non-standard choices in summary statistics)
• Organizational
  • Cursory validation (stakeholders sign-off too early due to lack of effort, too high costs, overreliance on opinion of experts, overreliance on opinion of vested managerial interests)
  • Corrupt and perverse selection (dataset selected specifically validate a pre-disposition or conviction on the part of experts and/or management)
• Maintenance / Re-validation
  • Failure to iterate / review periodically (in particular when there is no plan stipulated for periodic review and maintenance of the model – no ongoing ownership)

3.       IMPLEMENTATION:  verification – the risk of faulty operationalization

• Technical
  • Calculation errors
  • Programming errors
  • Software errors
  • Data implementation errors
• Operationalization
  • General poor operationalization (errors in design and/or specification creep into use)
  • Overreliance (overextension of model use beyond intended scenarios – the ‘magic oracle’ or ‘black-box’ syndrome)
  • Failure to use effectively (although accurate, system not attached appropriately to organizational decision making processes)
• Conclusions / inference
  • General errors of theory (improper theoretical framing such as implying causation where there is only correlation)
  • Type I Errors (incorrect rejection of a correct null hypothesis, leading to failure to refute an incorrect hypothesis and adoption of significance where there is none)
  • Type II Errors (failure to reject a false null hypothesis, leading to acceptance of significance where none is justified)

* This list is a first attempt and feedback is very welcome:  I will update and annotate this listing based upon suggested additions to and revisions to this list.  The final article will propose categories of approaches for dealing with business analytics model risk.  Email:  webmaster@sark7.com

End of article 4 of 5

Link to next article in series (5 of 5): pending June 2013

Link to introductory / header article (0 of 5)

———————————————————————————–

BLOG POSTS / ARTICLES OF INTEREST

Data Scientist: Bias, Backlash and Brutal Self-Criticism:  http://www.ibmbigdatahub.com/blog/data-scientist-bias-backlash-and-brutal-self-criticism

Careful: your big data analytics may be polluted by data scientist bias:  http://gigaom.com/2013/05/04/careful-your-big-data-analytics-may-be-polluted-by-data-scientist-bias

The hidden biases in big data:  http://blogs.hbr.org/cs/2013/04/the_hidden_biases_in_big_data.html

The dictatorship of data:  http://www.technologyreview.com/news/514591/the-dictatorship-of-data/

Evidence-based versus intuitive decision making:  http://blogs.hbr.org/hbr/mcafee/2010/01/the-future-of-decision-making.html

REFERENCES

Ansoff, H. I., & Hayes, R. L. (1973). Roles of models in corporate decision making. Paper presented at the Sixth IFORS International Conference on Operational Research, Amsterdam, Netherlands.

Balci, O. (1998). Verification, Validation and Testing: Principles, Methodology, Advances, Applications, and Practice. In J. Banks (Ed.), Handbook of Simulation. New York: John Wiley & Sons.

Derman, E. (1996). Model Risk. Quantitative Strategies Research Notes. Goldman Sachs. http://www.ederman.com/new/docs/gs-model_risk.pdf

Hubbard, Douglas W. (2009). The Failure of Risk Management: Why It’s Broken and How to Fix It. John Wiley and Sons: Kindle Edition.

Morini, Massimo (2011). Understanding and Managing Model Risk: A Practical Guide for Quants, Traders and Validators (The Wiley Finance Series). Wiley: Kindle Edition.

Sargent, R. G. (1996). Verifying and Validating Simulation Models. Paper presented at the 1996 Winter Simulation Conference, Piscataway, NJ.

Shannon, R. E. (1975). Systems Simulation: The Art and Science. Englewood Cliffs, NJ: Prentice-Hall.

Following from article 2 of 5 on Business Analytics Model Risk

Link to introductory header article (0 of 5)

In a the first article in this set on business analytics model validation (http://sctr7.com/2013/03/14/model-risk-when-is-a-business-analytics-model-validated/), it was proposed, in summary, that:

1) business decision models are becoming increasingly complex due to the co-emergence of technology, globalization, communication, and competition factors;

2) growing complexity in business decision models is increasing inherent difficulties associated with model validation, particularly in terms of coordinating consensus and orchestrating clear communication;

3) growing decision model complexity leads to a ‘squeeze’ in which greater numbers of stakeholders are involved, but fewer individuals are able to conceptually grasp the entire model;

4) exhaustive business model validation is not, in any case, methodologically possible, validation being essentially a process of developing organizational consensus that the model is ‘good enough’; and

4) due to these factors, therefore the general expressed need for better process-focused structure in organizational processes related to model design, review/validation, and implementation.

complexity

Citing Derman, who speaks in terms of financial model risk, but whose meaning we can apply to broader models: “a model is always an ‘attempted simplification of a reality’, and as such there can be no true or perfectly realistic model. But realism and reasonableness, coupled with simplicity, must remain crucial goals of a modeller, and their lack creates model risk.” (Morini, 2011).  Derman’s assertion, together with the above set of proposals and conclusion, imply the need for better structure in organizational processes associated with model design, validation, and implementation.  However, this can only come via a finer-grained understanding of the components of business analytics models themselves.

Since we have established (in article 2) we are concerned about business analytics model risk, beyond exclusive financial industry model risk, it is appropriate more closely define the scope for business analytics models.  The particular models of interest to business analytics (BA) are those which seek to characterize large, complex systems, often aggregating assessments of technical, economic, and behavioral phenomenon.  Such models are of increasing concern to BA practitioners as businesses seek to characterize larger and more complex systems.  Sustainability, global supply chains, outsourced workforces, multi-stakeholder collaboration, and intricate financial engineering are examples of phenomenon driving increasingly complex models.  Such models also typically necessitate validation by broader and more diverse groups of organizational stakeholders.

Concerning the functional specification of business analytics models, we mean the particular functional area of a business where the model proposes to guide decision making.  Figure 1, below, characterizes the trend to embrace increasingly complex sets and combinations of phenomenon in order to address increasingly intertwined business problems:

Interconnections

Figure 1:  Business analytics models increasingly embrace complex inter-systemic dynamics

The scope of BA models is expanding such that they frequently cross functional domains.  Thus, operations planning models typically involve financial factors and sales projections.  Likewise, a financial planning analysis will look at production factors as related to shifting demand

In highly focused implementations, models may be quite self-contained and discrete, applying to a particular domain.  For example, in financial derivatives trading there may be a particular set of algorithms set-up to assess and react to specific boundary conditions in particular markets.  Such models can be tested within their domain against historical data to achieve some understanding of pragmatic retrospective reliability.  Still, as has been the case in recent trading debacles, there are always circumstances in the market where hidden assumptions outside the model invert and cause monolithic misjudgment (i.e. a market collapse causing massive correlation between instruments as liquidity dissolves and market panic set in).  Even discrete financial trading models sit on top of more complex macroeconomic systems, which connect to political and demographic systems via obscure and complex dependencies.  In this sense, self-contained and discrete models can be useful, but are wishful in disregarding larger systemic connections which can cause ‘black swan’ events.

BA models, although they may incorporate a broad range of phenomenon, typically aim to reduce measurement to financial terms. Net Present Value (NPV) assessment is a typical standard for strategic business analytics modeling, as it allows like-to-like comparisons across projects.  However, to the degree BA models frequently attempt to forecast and predict, they may integrate a broad variety of phenomenon into a central NPV model.  For instance, an NPV assessment of a power plant project must countenance highly volatile, highly integrated long-term factors, such as:  perturbations in the price of coal, production output ranges, operational and maintenance overhead, market competition, electricity demand, political factors (i.e. tax policy changes, regulatory impositions, etc.), and the shifting of broad macroeconomic factors (i.e. interest rates, inflation, market disruption).

These types of broad, aggregative models are known as techno-economic models, in that they attempt to undertake broad, unified financial assessments incorporating highly uncertain economic and technical phenomenon.  Increasingly these models are being extended into another dimension: behavioral.  This is the case, for instance, when consumer behavior, market size in relation to marketing efforts, and the effects of competition are assessed.  As well, increasingly the behavioral aspects of financial markets are a topic of interest in complex models.  In the wake of the financial crisis, an assault on the traditional assumption of the ‘efficient market’ has given way to attempts to assess behavioral dynamics in markets.  Such models can be called techno-economic-behavioral.  Significantly, although it may be possible to validate discrete components of such complex, multi-component models, validating all sub-components does not imply a validated aggregate model (Balci, 1998).

Such composite models introduce model complexity, a unique risk factor in itself as a composite of linear systems quickly becomes nonlinear (discrete and relatively predictable systems become highly uncertain).  Sources and aspects contributing to model complexity is thus worthy of an attempt at categorical treatment.

SOURCES OF MODEL COMPLEXITY (NON-EXCLUSIVE / OFTEN OVERLAPPING)

1.  Stakeholder Complexity:  as models become more complex, they often encompass broader swaths of functional and systemic parts of the business.  This naturally leads to broader groups of stakeholders being involved.  Stakeholder involvement can originate from combinations of roles and interests:  manager / owners of affected resources, vested decision participants, area experts (i.e. engineers, market specialists, technical experts), targets of analysis (i.e. outsourced labor, 3^rd party companies, interested customers), or data providers (i.e. owners or providers of data).  To ensure robust models, some attempt must be made to map and involve interested parties in the process of model design, testing, and implementation.

2. Probabilistic Complexity: as more complex models are generated, a range of variables with associated uncertainty typically aggregate / are centralized in models. For convenience, here we use probabilistic complexity to encompass both raw uncertainty as well as probabilistic factors (although these are typically distinct and are treated separately in predictive analytical models).  The aggregation of multiple uncertain variables in models leads to nonlinear bevavior in predictive models.  Care must be take in terms of segmenting uncertain / probabilistic variables (i.e. distinguishing a raw unknown such as chance of a legal suit from probabilistic factors such as price uncertainty based on historical analysis).  Also, when aggregated, care must be taken in terms of the relative interaction of uncertain variables in the aggregate model.  Sensitivity analysis can help to rationalize and understand the aggregate behavior of multiple uncertain variables (i.e. tornado charts, ranking based on simulation).  Often a particular uncertain variable will subsume or dominate.  For example, currency exchange rate uncertainty may completely overwhelm other variables in a model.  Once tracked and accommodated (i.e. currency hedging plan put in place), a particular stochastic variable can be demoted in the model and other factors can be dealt with (i.e. insured, hedged, offset, retained, etc.).  The like-to-like treatment of multiple uncertain variables in an aggregate model should be dealt with in a methodical fashion which both examines the uncertain variable in isolation and then treats it as a relative component in a composite model.

3. Inter-systemic Complexity:  as models become more complex, they typically involve the interconnection of multiple systems.  This factor overlaps with the topic of probabilistic complexity, but can be distinct in terms of relatively simple (linear) sub-models being linked which creates aggregate non-linear dynamics.  For example, a complex manufacturing planning model may involve a sub-model examining the changing price of a commodity on the open market, changing dynamics in customer demand, plant production capacity, capital planning, personnel planning, and currency and interest rate fluctuations.  By aggregating a number of discrete sub-models into a master model, discrete elements may soon generate an aggregate an inherently nonlinear model in which: 1) there is no single optimal solution (i.e. there are only ‘best guesses’ and ongoing orchestration), and 2) relatively simple perturbations in sub-variables lead to highly chaotic effects in the macro system. A characterization of each sub-system must be refined enough that each factor is properly characterized, but attention must be dedicated to aggregate model ‘manageablity’.  In interlinking and characterizing multiple sub-systems in an aggregate model, it is essential to pay attention to two issues:  1) do not overwhelm the model with a ‘spaghetti’ representation of multiple variables co-varying across an unmanageable model, and 2) care and attention must be dedicated to understanding where the aggregate model may indicate the need for constraints-based management (i.e. Goldratt’s Theory of Constraints http://en.wikipedia.org/wiki/Theory_of_constraints ).  An example of an otherwise simple chain of systems which creates aggregate complex behavior (and can quickly lead to sub-optimal value destruction) is the classical bullwhip or whiplash effect (http://en.wikipedia.org/wiki/Bullwhip_effect):  i.e. creating volume-based price incentives for suppliers who pass on discounts to customers in the context of a sales-bonus driven culture can lead to regular stockouts and shortages which damage supplier confidence and customer satisfaction.  Such systems often are better served by establishing a central constraint, such as a regional warehouse that mandates ‘everyday low prices’ in order to optimize sales and to regularize stock handling.

4.  Functional Complexity:  as per Figure 1 above, functional model complexity involves mixing and matching functional business disciplines into composite models to gain insight.  As each domain contains embedded practices, theories, and assumptions, crossing domains in composite models introduces interdisciplinary complexity.  Spanning expert domains can introduce the risk of integration problems, comprehensiveness, mis-matched underlying assumptions, misunderstandings related to terminology and jargon, and long-term maintenance issues.

5.  Methodological Complexity:  composite methods (i.e. mixing statistical analysis, simulation, and decision trees).  This introduces risk related to the boundary conditions and assumptions of particular methodological techniques.  For instance, mixing linear and nonlinear methods for forecasting treats and views data in very different ways.  Multiple techniques also reduces composite model simplicity, introducing issues related to interpretation as well as long-term maintenance.

6.  Technical Complexity:  composite systems and technologies for handing and processing data to gain insights.  Somewhat self-explanatory, complexity can be introduced in terms of the technical handling of data from cleansing to insight as it moves through multiple platforms and systems.  There may be assumptions and procedures embedded in the data cleansing step which are subsequently not treated properly in subsequent data analysis.  An example concerns selecting samples and the treatment of outliers.  This can be an issue particularly in large enterprise environments when many types of functional IT and analytics experts ‘touch’ the data from inception to results interpretation.  As well, model assumptions can become trapped or wrapped in ‘black boxes’ to the degree there is unclear ownership and a lack of documentation.

This represents an attempt to categorize particular types of model complexity.  Please let me know if you have suggestions for edits/additions and I will update this list as an ongoing reference.

End of article 3 of 5

Link to next article in series: categorizing business analytics model risk (article 4 of 5)

Link to introductory / header article (0 of 5)

REFERENCES

Ansoff, H. I., & Hayes, R. L. (1973). Roles of models in corporate decision making. Paper presented at the Sixth IFORS International Conference on Operational Research, Amsterdam, Netherlands.

Balci, O. (1998). Verification, Validation and Testing: Principles, Methodology, Advances, Applications, and Practice. In J. Banks (Ed.), Handbook of Simulation. New York: John Wiley & Sons.

Derman, E. (1996). Model Risk. Quantitative Strategies Research Notes. Goldman Sachs. http://www.ederman.com/new/docs/gs-model_risk.pdf

Hubbard, Douglas W. (2009). The Failure of Risk Management: Why It’s Broken and How to Fix It. John Wiley and Sons: Kindle Edition.

Morini, Massimo (2011). Understanding and Managing Model Risk: A Practical Guide for Quants, Traders and Validators (The Wiley Finance Series). Wiley: Kindle Edition.

Sargent, R. G. (1996). Verifying and Validating Simulation Models. Paper presented at the 1996 Winter Simulation Conference, Piscataway, NJ.

Shannon, R. E. (1975). Systems Simulation: The Art and Science. Englewood Cliffs, NJ: Prentice-Hall.

Following from article 1 of 5 on Business Analytics Model Risk

Link to introductory header article (0 of 5)

-          Benjamin Franklin ‘The Way to Wealth’ (1758)

For want of a nail…

The parable ‘for want of a nail’ casts back to the Middle Ages, having been linked to Richard III’s unhorsing and defeat.  Its subsequent use has become synonymous with a general observation concerning both the interconnectedness of all things and the rule of unintended consequences, especially in their joint manifestation otherwise known as chaos theory’s ‘butterfly effect’ http://en.wikipedia.org/wiki/Butterfly_effect.

We can consider this adage an apt and illustrative admonishment concerning the general risk of ‘poor models’ leading to poor decision making consequences.  Models, especially as encoded in enterprise IT decision support systems (DSSs), are, via the proliferation of analytics-driven management, fast becoming the crux of organizational decision making.

Here we address this phenomenon by attempting to catalog the distinct types of business analytics model risk factors faced by firms.  Based on the principle that we cannot manage what cannot be measured (or specified discretely), this article attempts to sketch a ‘catalog’ or categorization of issues and items related to business analytics decision model risk.  Whereas model risk is often discussed in the singular, in practice it is encountered in multiplicity: there are a number of both functional and methodological sources of model risk, some of which overlap, but which benefit from being broken out and distinguished separately to improve apprehension.

Model risk is “the possibility that a financial institution suffers losses due to mistakes in the development and application of valuation models” (Morini, 2011).  The finance industry version of the nail parable becomes something along the lines of:  ‘for want of a morning coffee the calculation was off, for want of a calculation the spreadsheet was wrong, for want of a spreadsheet the traders were lost, for want of the traders the market was lost, all for want of care for morning coffee’.

While discussed most hotly by banking and investment institutions, model risk applies equally, and is of equal concern to, broader enterprise decision making.  Firms from Wal-Mart to Disney to Jet Blue are ardent and growing users of analytics-based decision models to drive and improve their core businesses.  Although most all business models ultimately come down to, or imply, financial outcomes, models are deployed to address an array of problems, from optimizing machine utilization in an assembly line, to reducing customer complaints / increasing satisfaction, to targeting advertizing for particular communities, to making crucial personnel decisions.

Thus, while banking and high-finance has assumed the banner charge on model risk, the broad topic applies to business analytics decision models more generally.  High-finance discussions of model risk quickly, and rightly, descend into industry-specific minutia quite quickly: volatility smiles, payoff assumptions, capital structure arbitrage, and Libor swaptions.  Such topics are important and significant to high finance, but are comprehensive mainly to specialists and vested parties.

We propose here that model risk has a broader and growing extra-banking industry context: that of the increasingly analytics-driven, model-based management of business enterprise more generally.  Increasingly a broad variety of firms are adopting business analytics decision models to drive and mange operations, strategic planning, marketing, customer service, and personnel decisions, often deploying advanced technologies and complex decision frameworks.  Of concern is that increasingly complex models are being adopted with a poor context and understanding of potential risks trapped in the models.

A mere five years ago, ‘model risk’ was considered an obscure and pedantic topic.  One might imagine it formerly debated chiefly by odd, pipe-smoking cabals of Wall Street quants and bitter, ignored professors of epistemology from small, unpronounceable private universities.  How karma has swiftly changed us all into ruddy-cheeked enthusiasts of modeling minutia!  The spectacular failure of a rogue’s gallery of key financial and macroeconomic models has thrust this topic to center stage.  One might easily envision Sergey Brin and Obama pontificating on the topic with Jamie Dimon in Davos.  The U.S. Mortgage Bubble, collapse of Lehman Brothers and Bear Sterns (among others), subsequent Global Financial Crisis, and the continuing debate concerning how to structure a staged recovery has been, at its core, a history of ‘models gone wild’: the good, the bad, and the ugly.

The current boisterous debate concerning government debt levels and economic stimulus illustrates quite graphically the pitfalls of model risk:  http://www.economist.com/news/finance-and-economics/21578704-mudslinging-between-economists-distraction-real-issues-dismal.  Economists Carmen Reinhart and Kenneth Rogoff previously had been lauded by austerity-pushing politicians worldwide.  Their research, based on statistical analysis claiming to equate high government debt levels with slowing economic growth, became a rallying cry for pro-Hayek / anti- Keynesian calls for smaller government, spending cuts, and lower public debt levels to climb out of the economic crisis.  In democratic governments populated by tax and debt-shy elected officials, this research provided a convenient excuse to slash government programs and investment instead of enacting stimulus measures.

The apparent ineffectiveness and pain inflicted by harsh austerity measures, particularly in the EU, led to a review of the research model.  It turned out several errors were revealed: 1) a coding error excluded relevant data from the sample, 2) relevant data was culled from the sample, and 3) a questionable method for weighting historical figures was utilized. http://blogs.lse.ac.uk/impactofsocialsciences/2013/04/24/reinhart-rogoff-revisited-why-we-need-open-data-in-economics/ .

Beyond this, revisiting the Rogoff-Reinhart research has highlighted a common ‘model risk’ factor:  the potential for incorrectly conflating demonstrated correlation with causation (i.e. assuming that because two phenomenoncoincide, that one ‘causes’ the other).  Their research implied that the seeming correlation between high public debt and slower growth indicated that raising public debt caused slower economic growth.  However, it may be that slower economic growth simply often co-occurs frequently with higher public debt as governments borrow in an attempt to stimulate the economy.  It does not necessarily follow that rising public debt is the cause of, as much as an accompaniment of, slower growth.

While Rogoff-Reinhart evidenced both technical and conceptual errors, it is the latter type which is typically the more subtle and thus dangerous risk.  This is the risk, essentially, of inflicting faulty theory, via assumptions, into the model and/or onto interpretations of model results.  As Morini espouses in Understanding and Managing Model Risk:  “model assumptions, not computational errors, were the focus of the most common criticisms against quantitative models in the crisis, such as ‘default correlations were too low’ ” (2011).

It is not without due cause that the topic of model risk topic has been thrust from the bowels of intelligentsia boiler rooms to the relative patter of market-watch chat shows.  Avoiding alarmism, the stakes are high.  Firstly, while some rather poor decisions have been made already on the basis of broken models, more could yet follow.  At a fundamental level, quite dramatic errors have been made over the past two-and-a-half decades due to faulty decision making processes which at some level rested upon model-based procedures: the Challenger and Columbia space shuttle disasters, the collapse of Long-Term Capital Management, derivatives-based investment melt-downs, the Dotcom investment bubble, intelligence failures surrounding 9/11, the lead-up to the Iraq War, friendly fire incidents in military theaters, the Hurricane Sandy disaster, and numerous recent trading scandals.  Understanding and counteracting analytics model risk seeks to avoid errors of decision making which have outsized consequences.

Secondly, there is an obverse scenario possible: given the increase in analytics model complexity and the possibility of swelling errors, there is the danger of a rejection of the advanced analytics paradigm, of ‘throwing the baby out with the bathwater’.  There are powerful new tools and techniques for conducting advanced data analysis.  However, it may occur that business leaders ultimately abandon the effort to manage complex decision making models and processes for their ungainliness and propensity for error when not properly validated.  A recent post addressed this topic: http://sctr7.com/2012/12/27/decision-management-hitting-natural-human-limits-and-what-to-do-about-it/

Implicit in the concern of rejecting advanced analytics is the notion that a decision model has marginal declining utility as the efforts dedicated to model management (principally design, validation, and implementation) increase as corresponding value decreases (see Figure 1 below). The danger is that organizations deem complex decision making too unmanageable, too costly, and too risky, and therefore retreat to a new ‘Dark Age’ driven by traditional intuition-based management.  Such a retrenchment to traditional intuition-driven, top-down management paradigms contains its own danger, as addressed in another recent post: http://sctr7.com/2013/05/19/the-once-and-future-king-is-anglo-saxon-business-culture-its-own-worst-enemy/

model cost

Figure 1: Value trade-off in model overhead (Ansoff & Hayes, 1973; Balci, 1998; Sargent, 1996; Shannon, 1975)

Given these risks, the following article will look more closely at issues related to model scoping, after which a working categorization of business analytics model risks will be offered.

End of article 2 of 5

Link to next article in series: business analytics model scoping and complexity (article 3 of 5)

Link to introductory / header article (0 of 5)

REFERENCES

Ansoff, H. I., & Hayes, R. L. (1973). Roles of models in corporate decision making. Paper presented at the Sixth IFORS International Conference on Operational Research, Amsterdam, Netherlands.

Balci, O. (1998). Verification, Validation and Testing: Principles, Methodology, Advances, Applications, and Practice. In J. Banks (Ed.), Handbook of Simulation. New York: John Wiley & Sons.

Derman, E. (1996). Model Risk. Quantitative Strategies Research Notes. Goldman Sachs. http://www.ederman.com/new/docs/gs-model_risk.pdf

Hubbard, Douglas W. (2009). The Failure of Risk Management: Why It’s Broken and How to Fix It. John Wiley and Sons: Kindle Edition.

Morini, Massimo (2011). Understanding and Managing Model Risk: A Practical Guide for Quants, Traders and Validators (The Wiley Finance Series). Wiley: Kindle Edition.

Sargent, R. G. (1996). Verifying and Validating Simulation Models. Paper presented at the 1996 Winter Simulation Conference, Piscataway, NJ.

Shannon, R. E. (1975). Systems Simulation: The Art and Science. Englewood Cliffs, NJ: Prentice-Hall.

LINK TO HEADER ARTICLE

LINK TO PREVIOUS ARTICLE IN SERIES (2 of 7)

3.  How does change management work?

At the core, change management sees the organization as a network of interconnected interests, both shared and conflicting. An organization can be viewed as being composed of actors who populate both formal and informal coalitions.  While superficial change can be easily realized by changing the ‘official’ organizational structure (i.e. the ‘org chart’), the ‘deep culture’ of an organization is more difficult to shift. Composing motivation, identity, drive, sympathies, social contracts, and tacit cooperation, ‘deep culture’ is the core of change and where initiatives succeed or flop.

Change can be implemented at the formal level by altering ‘org chart’ groups and roles.  This is the stereotypical ‘chess game’ expected in most reorganizations. Senior leadership announces changes are underway, managers enter into tense and cloistered negotiations, rapid ‘horse trading’ ensues, and a tense period of waiting follows, after which senior leadership descends from the mountain with a new covenant, specifying new groups and reporting relationships.

Lost Horizon I by Antony Gormley

Following the reorganization some benefit and ascend, at least on paper, with more reports and top-line power, while others descend, losing status, or are ‘downsized’ outright.  A lexicon of buzzwords surrounds the process to make those involved more comfortable, but the simple truth is that some are promoted, some demoted, and others are fired. A literal change has taken place in that the formal relationships in the organization have been shaken up.

However, lasting change requires ‘hearts and minds’, reconfiguring people’s basic motivations and identification with various informal coalitions.  This implies shifting the internal contract employees perceive having with their organization and management.  The much cited work The Heart of Change, by change management guru John Kotter, recognizes that change initiatives succeed or fail largely depending on the emotional and attitudinal disposition of key actors in the change process. Whereas the org chart can be reconfgured, people’s attitudes and dispositions are not so quickly or easily shifted.  The risk in reorganization is of achieving a ‘Frankenstein department’, a reconfigured chimera that lives anew, but which otherwise ignores the point and goals of the change.  Such recomposed groups shamble forward in an undead state, largely oblivious to the outside world, mixed and damaged.

To avoid a stillborn reorganization, Kotter proposes digging deep to change the underlying social connections and perceptions.  ‘Hearts and minds’ change management contains a notion of a guiding or leading coalition, key thought leaders who are assembled to publicize and motivate the change program.  Likewise, there are terms and categories for those involved: change advocates, early adopters, resistors, and passive resistors.

These actors inhabit varying positions of importance in the organizational network.  That is, a resistor who controls key processes and/or is a heavy influencer of others is considered a central target for change, either via intense negotiation or more harshly, if viewed as an intransigent ‘counterrevolutionary’. A change or resistance leader is not necessarily an agent of formal power: they may embody key information or expertise which is difficult to transfer, or may overtly or covertly control a step in a key process, or have undue influence over others in the organizational network due to some combination of seniority, expertise, knowledge, ethical standing, or charisma/respect.

The Kotter-driven change program initially seeks to ‘unfreeze’ the organization, breaking both formal and informal ties in the organizational network.  After this, change leaders communicate and clarify the change goals and steps.  Once explicit and implicit ‘buy-in’ is demonstrated, the change managers re-freeze, or re-structure the organizational network in a new configuration.  Both formal and informal structures are thus targeted to drive the new processes and procedures forward.

Kotter thus espouses having a dramatic start to the change process, highlighting the driving need for change and the dire consequences of not changing.  Once ‘shaken-up’, expectations surrounding the changed organization are communicated.  This includes expecting key participants to voice their commitment and ‘buy-in’.  After it is felt that an alteration of the basic compact between participants and leaders has been altered, the organization is ‘re-frozen’, or recomposed into a steady state to proceed forward.

This basic framework guides most change management initiatives, often driven as a concerted, structured effort managed by a collaboration of senior management, an indoctrinated ‘guiding coalition’ (often emerging middle managers and key experts who buy into the goals of the change initiative).  Typically, a team of consultants specialized in change programs is centrally involved.  Particularly where ‘downsizing’ is expected, the involvement of consultants offers a convenient third-party to deal with the stickier issues of who goes and who stays.

The positive aspect of the Kotter-associated change initiative is that it addresses and attempts to deal with the underlying socio-structural factors involved in a change program.  Recognizing that the organizational structure is only part of the story, ‘hearts and minds’ change seeks to become involved in social engineering.  In particular, there is an attempt to consciously reframe the perceived compact, the set of social agreements (both tacit and explicit), between employees and the organization.

However, Kotter-driven change has not been a panacea, and typically reorganizations still struggle despite such deeper intervention.  The author was involved in a large corporate reorganization where Kotter was involved, and the results ultimately were quite disappointing on most quantitative measures (cost overruns, decline in employee productivity, lower retention rates, costly failed projects, etc.).

Since authoring ‘The Heart of Change’, Kotter has addressed shortcomings and ‘misperceptions’ in a mea culpa Harvard Business Review (HBR) article entitles ‘Leading change: why transformation efforts fail’ (http://hbr.org/2007/01/leading-change-why-transformation-efforts-fail/ar/).  The article also admits that a change program is not a one-size-fits-all gambit.  While such admissions and retrenchment is instructive, it also reveals cracks and fissures in the change management industry.  There is a larger admission that organizational change is inherently difficult and that even the best laid plans often go astray.

Beyond this, critics have emerged to take aim at the traditional principles and processes of reorganization itself.  Some question, cynically, the logic of continually reconfiguring the organizational structure, commenting that at best the results are to shake up the organization, but at worst it is an opportunity to constantly prevent any type of effective coalition to develop effectiveness.

A recent HBR article by Marcia Blenko of Bain & Company, ‘The decision-driven organization’, takes square aim at the penchant for shifting organizational structures while ignoring central factors associated with effectiveness (http://hbr.org/2010/06/the-decision-driven-organization).  Blenko states: “we believe that this failure is rooted in a profound misunderstanding about the link between structure and performance. Contrary to popular belief, performance is not determined solely by the nature, scale, and disposition of resources, important though they may be… A corporation’s structure, similarly, will produce better performance if and only if it improves the organization’s ability to make and execute key decisions better and faster than competitors.”

The proposal from Blenko is that traditional change management often focuses on the wrong things, fetishizing the ‘toy soldier’ mentality of shifting boxes and lines around an org chart.  Rather, it is proposed that an organization should focus foremost on decision effectiveness, a process which may be connected to the org structure, but has more to do with organizational architectural attributes:  the location of decision making in the organization, access to data, information, and knowledge, incentives which propose to reward decision effectiveness, and assessment systems which specify how results will be measured.

Such a purview has more to do with Management Control Systems (Merchant & Stede, 2003) and organizational architecture, the HR discipline of aligning interests with roles.  It is thus conceivable to undertake an organization in which no formal reporting relationships change, but in which the organizational architecture shifts radically in order to reconfigure decision making and incentives.  This is perhaps the true ‘heart of change’ in that it attempts to shift behavior based on targeting rights, roles, and incentives, rather than ‘toy soldier’ empire building and management egos.

Here we get much closer to the topic of ‘analytics culture’ as this is concerned centrally with decision effectiveness.  It is proposed this adopting analytics culture is a type of organizational change, but that traditional ‘org chart’ reorganization misses the point.  Adopting organizational culture should attempt to improve decision effectiveness by co-locating decision making power with access to analytical insight.  This realigning of decision making can be ‘super-charged’  by clarifying the assessment scheme (metrics) and rewards for achieving specific benchmarks (as associated with and attached to the recomposed decision making rights).

We are then left with the notion that change management should be focused on decision processes more than organizational structure.  As well, we acknowledge that decision processes may be staged in both formal and informal networks within the organization, that a decision is not a simple linear path, but a circular process which involves coalitions of stakeholders.  It is this topic we will investigate in greater detail in following articles.  Firstly, however, the next article will take a deeper look at why reorganizations typically fail, and how an understanding of social networks can prevent such shortfalls.

REFERENCES

Blenko, M. W., Mankins, M. C., & Rogers, P. (June 2010). The decision-driven organization. Harvard Business Review, June 2010, p 54 – 62. Last retrieved May 5th, 2013 from http://hbr.org/2010/06/the-decision-driven-organization

Kotter, J. P., & Cohen, D. S. (2002). The Heart of Change. Boston, MA, USA: Harvard Business School Press.

Kotter, J. (January 2007). Leading change: why transformation efforts fail. Harvard Business Review. January 2007. Last retrieved June 2^nd, 2013 from http://hbr.org/2007/01/leading-change-why-transformation-efforts-fail/ar/

Kotter, J., & Schlesinger, L. (2008). Choosing strategies for change. Harvard Business Review. July – August 2008.

Merchant, K. A., & Stede, W. A. V. d. (2003). Management Control Systems: Performance Measurement, Evaluation and Incentives. London: Prentice Hall.

END OF PART 3 OF 7 IN A SERIES ON ADOPTING ANALYTICS CULTURE

LINK TO NEXT ARTICLE IN SERIES (4 of 7)

LINK TO HEADER ARTICLE

LINK TO HEADER ARTICLE

LINK TO PREVIOUS ARTICLE IN SERIES (1 of 7)

2.  Is change management effective?

When organizational change initiatives are undertaken, there is a common assumption that, after some struggling, the firm will improve.  With reorganizations there is often an assumption that the resulting structure will be progressive, or otherwise more efficient, ‘aligned’, or generally improved. This progressive bias assumes that when groups set-out purposefully to re-invent, beneficial adaptations and improvements naturally result. However, from the perspective of broader human history, many revolutions and political transitions result in steps backwards to states of greater disorganization, confusion, and waste.  Likewise, change management can easily result in less effective organizations.

Change Management

Change management, although seemingly omnipresent in modern organizational life, has a checkered track record.  A recent Bain & Company study on 57 corporate reorganizations “found that fewer than one-third produced any meaningful improvements in performance.  Most had no effect, and some actually destroyed value” (Blenko et al, 2010).  In the book Cracking the Code of Change, it is asserted that 70% of all change initiatives fail.

To complicate matters, indications are that today’s workers are fatigued by, skeptical of, and, often, ‘reflexively passively-resistant’ towards corporate change initiatives.  In Managing Change, it is observed that “by now, the troops have been through so many of these programs that they’re skeptical.  Companies today are full of ‘change survivors,’ cynical people who’ve learned how to live through change programs without really changing at all”.

Change management, though maturing, is, despite its central role in modern organizational life, still largely an emerging, imperfect discipline.  The practice has the potential for facilitating great improvements within institutions.  However, based on current assessments, there are even odds for value destruction. The discipline, having a disappointing success rate, needs to improve its ability to drive change, especially when faced with challenging change programs such as adopting analytics maturity/culture and structured decision making practices.

Having established the need for change management (see article 1) to orchestrate the adoption of ‘analytics culture’, how can the pitfalls of change management best be avoided? In the next article we will take a deeper, closer look at why many change initiatives fail.  From there, subsequent articles will propose a specific method for using social network analysis (SNA), a type of analytics for understanding social interactions in organizations, to structure and improve change initiatives.

REFERENCES

Beer, M. & Nohria, N. (May 2000). Cracking the code of change. Harvard Business Review. May – June 2000.

Blenko, M. W., Mankins, M. C., & Rogers, P. (June 2010). The decision-driven organization. Harvard Business Review, June 2010, p 54 – 62. Last retrieved May 5^th, 2013 from http://hbr.org/2010/06/the-decision-driven-organization

Duck, J. D. (1993). Managing change: the art of balancing. Harvard Business Review. November – December 1993.

END OF PART 2 OF 7 IN A SERIES ON ADOPTING ANALYTICS CULTURE

LINK TO NEXT ARTICLE IN SERIES (3 of 7)

LINK TO HEADER ARTICLE

LINK TO HEADER ARTICLE

1. What does change management have to do with business analytics?

Along with feverish interest in business analytics (BA) and ‘Big Data’ has been an interest in how organizations can adopt ‘analytics culture’ to evolve what has been called ‘analytics maturity’ (Kiron and Shockley 2011; Kiron, Shockley et al. 2011). The notion of analytics maturity as an aspect of organizational culture recognizes that analytics skills and technologies can only drive value if the organization has a core orientation towards evidence-based decision making.

Cognitive Bias

Analytics maturity, which involves adopting evidence-based decision processes, challenges the traditional top-down management paradigm.  In the typical large commercial organization, decision-making is still tied largely to the intuition-based judgment of ‘expert-managers’. Decisions are driven by expert judgment and often rely on intuition and heuristics (quick cognitive decision making shortcuts ‘baked into’ the structure of human mind).

While intuitive and heuristic decision making is not in-of-itself ‘wrong’, can be quite powerful, and will not necessarily lead to faulty decisions, recent research has explored the susceptibility of rapid, closed-loop decision making to well documented cognitive biases (http://en.wikipedia.org/wiki/List_of_cognitive_biases). As well, in complex organizational settings, the principal-agent problem exists, whereby incentives may be misaligned to the detriment of organizations and larger social interests (http://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem). The U.S. Mortgage Crisis and subsequent global Financial Crisis in particular evidenced both heuristic biases and agency interests gone awry.

Particularly where circumstances are inherently complex, where the amount of variables and overlapping systems overwhelm the capacity of individuals, even experts will fall back upon heuristics which may lead to sub-optimal decisions.  Nobel prize-winning psychologist Daniel Kahneman’s work ‘Thinking, Fast and Slow’ goes into detail concerning the susceptibility of  individuals and groups to leaping to sub-standard conclusions given complex problem sets (http://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow).

A number of case studies concerning well-publicized decision failures have similarly pointed out the traps individuals and groups easily fall into when presented with complex problems. Examples include detailed case studies related to, among others, the Challenger and Columbia space shuttle disasters, the collapse of Long-Term Capital Management, derivatives-based investment melt-downs, the Dotcom investment bubble, intelligence failures surrounding 9/11, the lead-up to the Iraq War, friendly fire incidents in military theaters, the Hurricane Sandy disaster, and numerous recent trading scandals. The Great Courses company offers an informative set of lectures by Professor Michael Roberto addressing ‘The Art of Critical Decision Making’ which examines some of these cases in detail: http://www.thegreatcourses.com/tgc/courses/course_detail.aspx?cid=5932

The past two decades have been replete with dramatic decision failures surrounding complex, interconnected systems and scenarios. It is asserted that such scenarios, involving immense social and technical systems which incorporate a multiplicity of actors and variables, are more and more the status quo for modern, global business. The complex of globalization, financial complexity, multi-stakeholder politics, large datasets, intricate computer-based systems, and the proliferation of communication channels via real-time digital media are all combining to make individual and intuition-based judgment more and more susceptible to dramatic failures when intuition leads to a reliance on snap heuristics rather that process-oriented decision making best practices.

Given the status quo paradigm of the expert-manager, it is asserted that adopting analytics-based culture, or evidence-based decision making, amounts to an organizational management paradigm shift. Proposing to shift the methods and basis for organizational decision making from power hierarchies and vested managers proposes to change power dynamics:  the organizational contract regarding access to information, decision rights, assessment systems, and incentive schemes.  Analytics culture, at root, proposes new organizational architectures and management control system schemes.  It is further asserted that change management, a recognized practitioner discipline for changing organizational culture, is the clearest mechanism to adopting such paradigmatic organizational change.

Thus the discipline and methods of organizational change management, by nature, sit at the center of attempts to adopt and implement analytics-based decision making. Analytics maturity implies that organizations make decisions primarily based upon evidence enhanced by analytical insight. Implicit are business processes which connect data/business analytics, communication/information sharing, and decision making. The struggle to adopt the mechanisms of analytics-based decision making necessitates organizational change management. However, as such mechanisms involve decision-rights and access to information, attempts to reengineer existing processes often meet organizational resistance.

The adoption of business analytics-based decision making involves streamlining decision processes via aligning techniques, technologies, and stakeholders.  Particularly in terms of re-aligning stakeholder power networks, adopting analytics-driven decision processes can easily encounter organizational resistance. Emerging technologies and methods put strain upon organizational leadership: new decision processes must be married to organizational structure and culture to be truly viable. Improved understandings of decision making processes in organizational networks can help to improve the robustness of business decision making change programs.

REFERENCES

Kahneman, D. (2011). Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.

Kahneman, D., & Klein, G. (2009). Conditions for Intuitive Expertise. American Psychologist, 64(6), 11.

Kiron, D., & Shockley, R. (2011). Creating Business Value with Analytics. MIT Sloan Management Review, 53(1), 10.

Kiron, D., Shockley, R., Kruschwitz, N., Finch, G., & Haydock, M. (2011). Analytics: The Widening Divide. MIT Sloan Management Review (Special Report), 21.

END OF PART 1 OF A SERIES ON ADOPTING ANALYTICS CULTURE: 1 of 7

LINK TO NEXT ARTICLE IN SERIES (2 0f 7)

LINK TO HEADER ARTICLE