Emerging Trends in Data Analytics


Business analytics is a practitioner movement uniting several disciplines to drive value-creating decisions from data.  Central disciplines include IT / computer science, statistics, data management, decision science, and scientific research methods.  Descriptive, predictive, and prescriptive approaches are often used to categorize particular methodological approaches, themselves derived from the fields of business intelligence, financial forecasting and econometrics, and operations management, respectively.  Diagnostics, dynamic visualization, and semantic analytics are particular supporting techniques.


Some claim that analytics and ‘big data’ have reached a hype inflection and that a denouement awaits.  This notion is reflected in the Gartner ‘technology hype cycle’, a reoccurring phenomenon associated with new innovations whereby overinflated expectations crash interest, followed by a pragmatic retrenchment.


The hype cycle observes that general and marketing-driven over-exuberance inflates expectations during the introduction of a new technology.  The subsequent denouement, in the form of unmet expectations leading to disappointment, over-adjusts.  However, in due course, provided the disappointment does not cause complete abandonment, there is a retrenchment in which the true value of the new technology or innovation is established and operationalized.  Often the retrenchment leads to subsequent waves of innovation, which duly are over-promoted in yet another cycle.

The most recent dramatic macro-example of this phenomenon was the Dot-Com boom followed by the Dot-Bomb bust.  The web did not disappear, as many doomsayers predicted during the 2001 market adjustment.  Rather, a retrenchment has occurred in which a refocusing on the practical value of web-based technologies has occurred.  The repurposing of web technologies to serve practical, in particular, commercial, goals is now a mainstay of the developed world, so much so that the rapidity and reach of the web is largely taken for granted as general infrastructure.  This has also led to secondary innovation waves:  social media, mobile, and the emerging internet of things (machine-to-machine internet communication), all of which will also likely disappoint, readjust, retrench, and re-emerge as per the hype cycle pattern.

One key aspect to second-wave web innovations is that they are generating increasing amounts of data which require analytics.  This has created an intense interest in data analytics, itself subject to the ‘hype cycle’ – an initial over-enthusiasm, followed by a denouement, and then a pragmatic retrenchment.  The subject of this post does not dwell on the reasons for the analytics hype, nor the valid critiques seeking to dampen expectations.  Rather, the intention is to raise several core emerging trends which underlie the analytics movement and, it is asserted, will be the foundation for the inevitable retrenchment.  This is a complicated proposition as the ‘analytics movement’, as it has been called, is not a single innovation, but a splintering of many innovative applications and methods for deriving value from data analysis.

While speculative in nature, the intention is to raise consciousness concerning long-term trends for the sake of practitioners, particularly for planners concerned with long-term strategy.  As always, feedback is welcomed, and insightful critique will lead to revisions to or additions to the proposed trends, proper credit being applied:

Twelve Emerging Trends in Data Analytics

The following twelve trends are asserted as the basis for the evolving data analytics ‘plateau of productivity’:

  1. Plumbers wanted: data management overhead demands professional data mungers
  2. Hardening models:  increasingly complex models require tighter approaches to diagnostics and validation
  3. The tunnel link: big data engineering and methodological approaches meet in the middle
  4. Change management to the fore: evidence-based decision-making requires management to contemplate new organizational forms
  5. Invisible architectures: enterprise architecture embraces systems management to forge a path through the mist of multi-systems complexity
  6. We’re not in Kansas anymore:  increasingly diffuse models requires a deeper methodological understanding of broader research paradigms
  7. Living with the paradox: coming to terms with irresolvable methodological quandaries
  8. Cyborg enterprise:  industrial-scale analytics ushers in the age of highly integrated, large-scale techno-organizational decision programs
  9. Not for everyone, but necessary none-the-less:  analytics as a service and outsourcing analytics as a function
  10. On-ramping AI: organizational operationalization as a step towards machine automation
  11. Emerging profession: professional computational decision engineers and AI stewardship
  12. Far-future: the birth of the Chief Meaning Officer – equal parts decision scientist, IT manager, storyteller, and organizational anthropologist

Trends one through three: video blog summary by Scott Mongeau

  1. Plumbers wanted: data management overhead demands professional data mungers

While we will see the advent of increasingly powerful tools to both manage and analyze large sets of data, the ever increasing volume of data besieging organizations will create increasing demands for specialized ‘data plumbers’.  This is to say that as much as we have network, infrastructure, and security professionals in most IT organizations, we will increasingly see the emergence of data management engineers as a distinct technical professional, a role more diverse and broader than the traditional data warehousing professional.

A recent New York Times article, ‘For Big-Data Scientists, ‘Janitor Work’ is a Key Hurdle to Insights’. attested to this increasing trend.  This was reinforced in a recent article in the INFORMS Analytics Magazine ‘Students, Professionals Need ‘Data Wrangling’ Skills’.  The observation is that increasingly large and diverse multi-format datasets demand more intensive approaches to managing and transforming data (see recent related post the demands data availability and quality are putting on professionals).

While some organizations already have data governance and database management professionals, increasingly data management engineering will emerge as a specialized discipline.  This will co-occur with the continuing maturation of specialized software tools for acquiring, cleaning, transforming storing, and retrieving data.  Data warehouses will increasingly not be enough, with data needing to be available on demand in a variety of formats and in flexible forms.  Facilitating this will be specialized data management engineers who ensure the smooth flow and processing of data, much as a hydraulic engineering ensures high-quality water is safely and efficiently provisioned in a municipality – a core infrastructure service.

  1. Hardening models:  increasingly complex models require tighter approaches to diagnostics and validation

The proliferation of analytical methods has seen the emergence of the notion of an ‘analytical process’ which manages an analytical model through the life cycle of inception, testing, development, operationalization, and maintenance.  As well, the continuing expansion of data variety, velocity, volume, and complexity raises methodological issues related to dataset selection and representation prior to analysis.

An earlier set of posts goes into detail concerning challenges associated with business analytics model validation – pointing out that with increasingly complex analytics models, the challenge of validation is increasingly organizational in nature.  The question of complex model validation connects to fundamental questions of organizational sensemaking, as covered in the post Analytics and Belief: The Struggle for Truth.

The projected trend concerns the adoption of formal methods to validate analytics models:  Business analytics model risk (part 1 of 5): when is a business analytics model ‘validated’?  The implication is that business analytics will begin formalizing approaches to decision model validation, reflecting more mature validation processes found in the finance industry and the discipline of systems engineering.


  1. The tunnel link: big data engineering and methodological approaches meet in the middle

The analytics movement has seen the growth of a host of facilitating technologies and methods.  For example, NOSQL, large volume storage and retrieval approaches (i.e. Hadoop), ensemble machine learning models, visualization, and semantic analytics.  Having evolved somewhat independently, such facilitating innovations will begin to merge, recombine, and transform into standard processes and patterns.

One key and fast emerging area of fusion involves merging advanced data engineering with data science methodological innovations.  By this, it is meant mixing the ‘plumbing’ of data engineering with the actual methods of data analytics such that common processes emerge as patterns, which themselves will be codified and embedded in commercial software offerings.

Sheer computational power, processing and transmission speed (exponentially increasing in rough synchronicity with Moore’s Law) coupled with high-volume storage and retrieval mechanisms (i.e. ‘the cloud’ / Hadoop / MapReduce), has enabled the ‘plumbing’ which supports the advance of big data analytics.  However, the advancement of the analytics movement has also necessitated improved data management (i.e. ETL) and methodological approaches (i.e. new algorithmic and machine learning approaches to detect and substantiate patterns in complex and/or large datasets).  Having matured somewhat independently (at least in terms of the general division in professional practitioner expertise), these two disciplines will increasingly begin to merge and evolve together, signaling the emergence of hybrid data plumber/scientists and increasingly codified supporting software tools.

Currently Hadoop, itself a finicky and low-level ‘toolkit’, is being improved by an array of commercial concerns developing layers to improve the management of big data storage and retrieval.  Similarly, new tools are emerging which streamline and automate the analysis of data (i.e. data selection, transformation, categorization, classification, prediction, simulation, and optimization).

With increases in data volume, variety, velocity, and complexity, standard relational database management storage and retrieval has been supplemented with an array of complementary and supplementary data engineering approaches.  The recent O’Reilly book ‘Agile Data Science’ by Russell Jurney espouses viewing big data management as a process, each utilizing particular technological and engineering solution approaches:  events -> collectors -> bulk storage -> batch processing -> distributed storage -> application server -> browser.  These steps have increasingly allowed for the acquisition, transformation, storage, and retrieval of large volumes of data at a rapid pace, the so-called big data approach.  Tools, utilities, and standards such as Hadoop, MongoDB, Avro, Pig, Casandra, and SPARQL allow for new approaches to big data management.

Likewise, the maturing of data science via disciplines such as machine learning, computational statistics, and graph mathematics has made available an array of increasingly powerful software-driven approaches.  Particular methodologies require data to be made available in particular formats.  However, increasingly data scientists also have a notion of a process, one in which a theoretical model is developed, tested, and implemented via a series of methodological treatments.  As an example, a marketing-focused data scientist may start by applying exploratory and unsupervised methods to develop an initial understanding of patterns and clusters resident in a large customer demographics dataset.  Having established clear patterns, the data scientist may then proceed to apply supervised methods such as machine learning and multi-regression analysis to substantiate correlative and/or causal connections between clusters and customer behavior, such as buying behavior associated with particular customer demographic groups.

Perhaps because data engineering and data science are themselves quite demanding areas of expertise, the combination of the two disciplines has thus far been quite pragmatic, with data engineers partnering with data scientists to ensure data is made available for various tests and analytical procedures.  It is easy to surmise that this trend of cooperation with continue and mature, to the extent that we will likely begin to see stronger formalization of data management processes with explicit data science methodological procedures in mind.

While this fusion has already begun, the level of engineering and expertise required to bring together big data management and deep data science is quite demanding, the tools and techniques are currently quite low-level, requiring a heavy investment in skills and technology development accessible only to the most mature and ambitious of analytics companies.  However, the latent demand for improved data processing with procedural analytical methods in mind will ultimately see the creation of more powerful software tools to drive integrated data engineering and analytics processes.  Some vendors such as SAS already market integrated toolsets for data management and analysis.  This trend will continue, with increasingly powerful suites merging advanced data engineering facilities with analytics studios.

Likewise, we will see professionals emerge who understand both sides of the technical divide, merging approaches to data plumbing in new and creative ways with analysis.  Indeed this is a growing necessity as large and complex sets of data can be selected, sliced, and transformed in multiplicitous ways.   For data science to sustain methodological credibility amidst growing critiques of fundamental methodological misunderstandings (i.e. the recent Science article critiquing Google Flu Trends), it must begin to formalize and improve the connection between dataset selection and representation with analytical methods, themselves being highly sensitive to issues related to selection and representation bias.

(to be continued…)

, , ,

About SARK7

Scott Allen Mongeau (@SARK7), an INFORMS Certified Analytics Professional (CAP), is a researcher, lecturer, and consulting Data Scientist. Scott has over 30 years of project-focused experience in data analytics across a range of industries, including IT, biotech, pharma, materials, insurance, law enforcement, financial services, and start-ups. Scott is a part-time lecturer and PhD (abd) researcher at Nyenrode Business University on the topic of data science. He holds a Global Executive MBA (OneMBA) and Masters in Financial Management from Erasmus Rotterdam School of Management (RSM). He has a Certificate in Finance from University of California at Berkeley Extension, a MA in Communication from the University of Texas at Austin, and a Graduate Degree (GD) in Applied Information Systems Management from the Royal Melbourne Institute of Technology (RMIT). He holds a BPhil from Miami University of Ohio. Having lived and worked in a number of countries, Scott is a dual American and Dutch citizen. He may be contacted at: webmaster@sark7.com LinkedIn: https://www.linkedin.com/in/smongeau/ Twitter: @sark7 Blog: sctr7.com Web: www.sark7.com All posts are copyright © 2020 SARK7 All external materials utilized imply no ownership rights and are presented purely for educational purposes.

View all posts by SARK7


Subscribe to our RSS feed and social profiles to receive updates.

10 Comments on “Emerging Trends in Data Analytics”

  1. Tim Negris Says:

    Enlightening, clear, and mercifully free of the double talk and blather that surround this subject in most quarters these days. Very well-written, too.


  2. R. Craig Reinsch Says:

    Scott – It would be great if you could expand on your description of what “Visualization” means in the first diagram, and maybe expand just a little on why it is floating up there off the main trail. I note that others have enjoyed the diagram, but never succeed in addressing this question.



  1. Business analytics model risk (part 1 of 5): when is a business analytics model ‘validated’? | BAM! Business Analytics Management… - July 15, 2014

    […] See also – recent post: 12 trends in data analytics […]

  2. Predictive policing: the brave new age of law enforcement analytics | BAM! Business Analytics Management… - September 5, 2014

    […] static and structured ‘reports’ – traditional descriptive Business Intelligence – to predictive, prescriptive, and semantic analytics […]

  3. 12 Emerging Trends in Data Analytics | Make Data Work - February 24, 2016

    […] Data Science Central shared its predictions for 2016. More predictions can be found here. In this article, we share Scott Mongeau’s predictions. The full version of this (long) article can be found here.  […]

  4. Tendencias Data Science para este año | Blog SoyData - tu recurso online sobre Big Data en español - March 1, 2016

    […] en el sector Data Science para este año 2016. Esta información la hemos recopilado del blog BAM! Business Analytics Management, Data Science Central e import.io Allá […]

  5. Ten Best Infographics of Big Data – Descriptive, Predictive and Prescriptive Methods – Aziyati Yusoff - July 19, 2017

    […] 4. Emerging Trends in Data Analytics by Business Analytics Management (BAM!) […]

  6. 12 Emerging Trends in Data Analytics – a. → all Story at a. - July 20, 2020

    […] Data Science Central shared its predictions for 2016. More predictions can be found here. In this article, we share Scott Mongeau’s predictions. The full version of this (long) article can be found here.  […]

  7. | CSDS portfolio sark7SCTR7: Data Science and Analytics - December 18, 2022

    […] Emerging Trends:Emerging Trends in Data Analytics  […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: