Machine learning is mentioned frequently in the media. How is it different from classical statistics? How is it similar?

We found it best to learn about this from Brian Caffo, a Ph.D. in statistics who also has expertise in computing algorithms. Brian is a professor of biostatistics at John Hopkins University.

Brian uses both machine learning and classical statistical procedures and covers the key characteristics of both.

First up: Machine Learning

  • User friendly
    • Easier to get started using it than classical statistics
  • Good predictions
    • ML algorithms contain many statistical procedures and therefore can yield good predictions
  • Automated
    • Very useful for online retail analysis, e.g., where obtaining the results with large datasets is important
  • Black Box
    • Except for adjustments needed, you don’t need to know the internal aspects
  • Overfitting is an issue
    • This is due to the inclusion of many parameters. But there are solutions to the overfitting problem.
  • Generalizability
    • As opposed to classical statistics where if the model is correct, you have generalizability, ML algorithms may work well with one dataset but not with another
  • Lots of Data
    • Work well on very large datasets. They aren’t applicable to small data sets.

Next, Classical Statistics

  • Parsimony
    • Key characteristic of classic statistical procedures. You want a simple model
  • Interpretability
    • Simple models lead to easier interpretability
  • Knowledge Creation
    • Historically, the simple, easily interpreted models have led to increasing knowledge
  • Inference
    • A major concern
  • Population
    • Another concern is connecting your model to the population
  • Generalizability
    • If you get a conceptual model that is true, you have generalizability
  • Little Data
    • Designed to work with smaller data sets. ML algorithms, especially using many parameters, are not applicable

You can view Brian’s video here.

In the very first part of Brian’s video, he mentioned his executive data science lectures where you can learn more about machine learning vs. classical statistics.

Here is the first video in that series discussing the topic. In it Brian uses several examples to show case the differences and illustrates what he would have done applying traditional statistical modeling.

  • Netflix Prize
  • Heritage Health Prize
  • Google Flu Trends

You can view the video here.  

Finally, here is the second video discussing machine learning vs. classical statistics. In it Brian lists three excellent references for more reading on the topic.

  • Larry Wasserman – Rise of the Machines
  • Leo Breiman: Statistical Modeling: The Two Cultures
  • D.R. Cox: A Comment on Breiman’s article
  • David J. Hand: Classifier Technology and the Illusion of Progress

You can view the video here.

Now, for those of you who are more familiar with machine learning, we can learn from a data scientist who also happens to hold a Ph.D. in statistics, Zach Kurtz, as he translates between machine learning and statistics.

In his article, Zach, based on his experience, compiles a table of common statistics and machine learning terms and the connections between them.

You can view Zach’s short article here.