The Big Picture About Big Data

Physician leaders and their organizations need a strong understanding of the benefits of harvesting and analyzing information about  patients — from cost savings to better individual treatment.

In his 2015 book Team of Teams: New Rules of Engagement for a Complex World, retired U.S. Army Gen. Stanley McChrystal describes the transformation a special task force underwent to battle al-Qaida in Iraq. “Driven by the necessity to keep pace with an agile enemy and a complex environment, we had become adaptable,” he writes. “We had fused a radical sharing of information with extreme decentralization of decision-making authority.”

Something similar is happening in health care. We are in the midst of an unprecedented data boom — genomic databases, electronic medical records and images, vital sign monitors, wearable devices, administrative claims and social media, among others.

Previously, health care leaders have relied on traditional methods — such as large, randomized, controlled trials, and expert opinion — to reach consensus on any topic. But the volume of data available today has rendered traditional analyses— using conventional software and hardware and statistical techniques — inadequate.


Big data is an evolving term used to describe massive volumes of information with the potential to be mined for business purposes. The information can come from numerous sources — such as sales records, experiment results or real-time sensors — and can be used to understand and respond to consumer behavior.

In a medical environment, it can be analyzed scientifically to determine, for example, the effectiveness of a new drug, an experimental procedure or a specific physical regimen.

Its name contrasts with the concept of “small data” — information whose volume and format can be analyzed by average users. A common expression: “Big data is for machines; small data is for people.”

“Big data” is collected and analyzed extensively by other industries — think: Netflix, Amazon and Google — to gain insights into consumer behavior and expectations. Health care is catching on, and leaders are calling for more innovative thinking and advanced strategies to harness and use this information to optimize their business and outcomes for their patients.

To be successful in this increasingly complex and competitive environment, organizations and leaders need to be just as adaptable as McChrystal’s special task force. Understanding, analyzing and putting big data into operation should be of paramount importance to senior health care leaders in every organization.

Theoretical Framework

Analyzing big data sometimes is described as a paradigm of four V’s: velocity, volume, variety and veracity.

Theoretically, advanced analysis of big data differs significantly from conventional tools of clinical epidemiology. In epidemiology, priority is given to defining causal mechanisms that can explain known events in the wider population. Analyzing big data is geared toward identifying correlations in the present (“the present truth”) in a locally stable sample (for example, the ICU population of a specific health care system). The analytical results of big data are inherently subject to continuous modification as new information is factored into the analysis to form “the new truth.”

Using big data requires shifting from the epidemiological emphasis on hypothesis to an information-centric approach. This means inductive reasoning deserves equal weight as deductive reasoning, which is employed in traditional research methodologies. Despite the stark differences in their approaches, one can argue that big data analytics and clinical epidemiology actually try to answer similar questions but on a different scale and with varying abilities. In fact, they can be seen as being complementary.

Big data analytics can help fill the void created by epidemiology’s inability to answer predictive (“what happens if I do something different?”) and prescriptive (“should I do something different?”) questions.

In other words, answers to many clinical questions cannot be gleaned from hypothesis generation and causal inference-centric analytical approaches such as controlled trials.

It’s likely clinical outcomes emerge from complex interactions among individual agents (genetics, environment, diet, occupational risks, etc.) — the so-called “complex adaptive systems” — as opposed to reductionist “cause-effect models.” For that reason, traditional techniques need to be supplemented by innovations in data science collection and analysis.1


Big-data tools can crunch unstructured data sets cheaply — a relative phenomenon in the universe of traditional epidemiological and statistical disciplines.2  Low-cost computers have been integrated into almost every component of patient care, such as intravenous pumps, ventilators, wearable devices and tablets, and it’s easy to extract, transform and load data from these disparate sources into relatively clean data sets.

However, it’s important to understand that “analytics” is not a single entity or program but a consortium of tools arising from the disciplines of computer science, engineering and mathematics.2 Big data can be thought of as repeated observations of a datum point or data points spread over time and space during day-to-day clinical care. Once established, data-gathering programs can potentially gather information on a permanent or ongoing basis.

MORE ON TECHNOLOGY: Articles on Tools and Strategies That Influence Physician Leadership

Advanced analysis of this information can help physician leaders and their organizations achieve the cherished goals of modern medicine — predictive, preventive, personalized and participatory patient care in a timely fashion.3 Many experts predict the use of big data will help physicians and organizations discover new models for calculating a person’s individual risk for developing disease.4  Certainly, the output can help the health care system transition into a “learning” entity characterized by constant and unimpeded exchange of information between a health system’s clinical research and its operations.

On a broader level, advanced analytical techniques could help usher in a new era of personalized drugs and drug-target discovery, facilitating the practice of precision medicine. Also, post-marketing evaluation of medicines could be facilitated by mining social networks and news media. Likewise, assessing an individual patient’s risk using big data (genomic as well non-genomic data) can lead to prescription of personalized health plans — getting us closer to the goal of personalized medicine.5

Applications and Challenges

Collecting and analyzing data is applicable across multiple domains of health care — including elucidation of disease mechanisms, new drug discovery, preventive care, clinical care, clinical operations and public health.6 Indeed, many experts believe big-data analytics can help reduce readmission rates, have a salutary effect on personal and population health, and steer the health care system toward a more patient-centered focus. In nonclinical application, analyzing big data can detect patterns of fraud and “revenue leakage,” and predictive modeling can be used to construct a crisper research and development pipeline, saving time and money.

That said, the term “big” does not necessarily mean “superior.” Data artifacts (“noise”) easily can creep into the magnitude of data that accrues over time.7 Likewise, privacy and ethical concerns are appropriate, considering how some data is collected — through social media posts, smartphones and wearable devices. And there’s concern about the validity of the analytic output, as evidenced by the failure of Google Flu Trends to predict the prevalence of flu in 2013.8 The online tracker, which debuted in 2008, used internet search data to predict influenza outbreaks; it was killed in 2015 after repeatedly producing dubious estimates.

Information generated by big data has the advantage of real-time display (“big answers”). However, corrective actions (such as clinical decision support systems) based on such output also need to be deployed quickly to reap the benefits. Some experts refer to the need for such solutions as “big opportunities.”2

Currently, big-data tools require intensive programming expertise to use. They will need to evolve into user-friendly interfaces to have broader, day-to-day application. Until then, some industry experts predict a rising demand for informatics-trained personnel with expertise in big data and advanced analytical methods. Without trained informaticists in place, organizations almost certainly will miss useful opportunities. Informaticists can help the big-data movement gather momentum by devising efficient means of organization and making them readily available.1 There also is a possibility organizations will miss the next revolution if collaborative research networks don’t evolve in the spirit of cooperation and advancement.

In all, health care systems that collect and analyze are likely to benefit from clinical outcomes, better population health management and reduced costs. It’s imperative that CIOs and CMIOs — as well as CEOs, CMOs, CNOs, and CFOs — have an understanding of everything that can be done with data. Although big data isn’t likely to replace traditional clinical epidemiology, its analysis can augment the decision-making process, filling voids where questions cannot be answered using existing methods.

Srinivas Mummadi, MD, FCCP, is chief medical information officer and the director of pulmonary and critical care medicine for Tuality Healthcare, based in Hillsboro, Oregon. Rakesh K. Pai, MD, MBA, CPE, is executive medical director for Oregon at Cambia Health Solutions in Portland, Oregon. Peter Hahn, MD, MBA, CPE, is chief medical officer and chief operating officer of Metro Health in Wyoming, Michigan.


  1. Krumholz HM (2014). Big data and new knowledge in medicine: the thinking, training, and tools needed for a learning health system. Health Affairs (Project Hope), 33(7), 1163–1170. hlthaff.2014.005
  2. Iwashyna TJ and Liu V (2014). What’s So Different about Big Data?. A Primer for Clinicians Trained to Think Epidemiologically. Annals of the American Thoracic Society, 11(7), 1130–1135. AnnalsATS.201405-185AS
  3. Gray EA and Thorpe, JH (2015). Comparative effectiveness research and big data: balancing potential with legal and ethical considerations. Journal of Comparative Effectiveness Research, 4(1), 61–74. cer.14.51
  4. Snyderman R and Yoediono Z (2006). Prospective care: a personalized, preventative approach to medicine. Pharmacogenomics. England.
  5. Chawla, NV and Davis DA (2013). Bringing big data to personalized healthcare: a patient-centered framework. Journal of General Internal Medicine, 28 Suppl 3, S660–5.
  6. Raghupathi W and Raghupathi V (2014). Big data analytics in healthcare: promise and potential. Health Information Science and Systems, 2(1), 3.
  7. Simpao AF, Ahumada LM and Rehman MA (2015). Big data and visual analytics in anaesthesia and health care. British Journal of Anaesthesia, 115(January), 1–7.
  8. Lazer D, Kennedy R, King G and Vespignani A (2014). Big data. The parable of Google Flu: traps in big data analysis. Science (New York, N.Y.), 343(6176), 1203–1205.

Topics: Technology Journal

Companies Need to Rethink What Cybersecurity Leadership Is
Attractive People Get Unfair Advantages at Work. Can Artificial Intelligence Help?
Transform your organization - Get leadership training from the experts!

Popular Articles


About Physician Leadership News

Now more than ever, physicians are leaders in their organizations and communities.

The American Association for Physician Leadership maximizes and supports physician leadership through education, community, and influence. We promote thought leadership in health care through our Physician Leadership News website, bimonthly Physician Leadership Journal and other channels.

We focus on industry leadership issues such as patient care, finance, professional development, law, and technology. Association announcements and news of association events can be found.

Send us your feedback at

Journal Submission Guidelines

AAPL's award-winning print publication, the Physician Leadership Journal, welcomes originally authored manuscripts for peer review that meet competency, formatting and preparation criteria. To review these guidelines and other information regarding submissions, click here.