From Data to Decisions: Measurement, Uncertainty, Analysis, and Modeling

CHE 379/384, Chris Mack, The University of Texas at Austin

 

Data Sets for use in this class:

Univariate data sets: Data_Sets_1.xlsx

Bivariate and multivariate data sets: Data_Sets_2.xlsx

Time series data sets: Data_Sets_3.xlsx

More bivariate and multivariate data sets: Data_Sets_4.xlsx

Data sets for Design of Experiments: Design of Experiments.xlsx

Body Fat.xlsx, bodyfat-reduced.csv, Model Building - Used Car Value.xlsx

 

Course materials, by class date for the Fall 2016 semester

Wednesday, August 24

Lecture 0: Introduction (10 min) - hardcopy of the slides: Introduction.pdf

Getting started with R:
Install R on your computer by going here.
Install RStudio on your computer by going here (there is a free version).
A script to use as an Introduction: Introduction demo.R
Some training videos: https://stat.utexas.edu/videos/r

Friday, August 26

Lecture 1: The Knowledge Hierarchy (22 min) - hardcopy of the slides: Lecture1.pdf

Read this this document: The Knowledge Hierarchy and the Data to Decision Process

Lecture 2: Data and Measurement (24 min) - hardcopy of the slides: Lecture2.pdf

Monday, August 29

Lecture 3: Data Example (11 min) - hardcopy of the slides: Lecture3.pdf

Lecture 4: Process Modeling (25 min) - hardcopy of the slides: Lecture4.pdf

Reading: NIST Statistics Handbook

Wednesday, August 31

Homework #1 (due in one week): HW1_Linear Regression in Excel.pdf; HW1_Data.xls

Lecture 5: Regression Review, part 1 (26 min) - hardcopy of the slides: Lecture5.pdf

Lecture 6: Regression Review, part 2 (31 min) - hardcopy of the slides: Lecture6.pdf

Class Summary Notes: Least-Squares Regression

Reading on matrix formulation for OLS: Penn State Course

Friday, September 2

Lecture 7: Appendix: Matrix Math (13 min) - hardcopy of the slides: Lecture7.pdf

Lecture 8: Regression Review, part 3 (14 min) - hardcopy of the slides: Lecture8.pdf

Lecture 9: Linear Regression in Excel (19 min) - Excel spreadsheet: Lecture9.xlsx

Bonus Lecture: Linear Regression in R (10 min) - Linear Regression.R

Anscombe's 1973 paper, Graphs in Statistical Analysis, and a spreadsheet of the data from that paper: Anscombe_Data.xls

Monday, September 5

Labor Day - no class!

Wednesday, September 7

Homework #2 (due one week from today): HW2_QQ Plots.pdf

Lecture 10: What is the Distribution of the Residuals? (16 min) - hardcopy of the slides: Lecture10.pdf

Bonus Lecture: Fun with Histograms (5 min)

Lecture 11: Q-Q and Normal Probability Plots (18 min) - hardcopy of the slides: Lecture11.pdf

Reading - Normal Probability Plots: http://www.itl.nist.gov/div898/handbook/eda/section3/normprpl.htm

Lecture 12: Normal Probability Plots in Excel: Lecture12.xlsx, Lecture12b.xlsx

Friday, September 9

Lecture 13: Testing for Skewness (16 min) - hardcopy of the slides: Lecture13.pdf

Lecture 14: Testing for Kurtosis (14 min) - hardcopy of the slides: Lecture14.pdf

Lecture 15: Performing Moment Tests in Excel (16 min): Lecture15.xlsx

Monday, September 12

Lecture 16: Shipario-Wilk Test for Normality (11 min) - hardcopy of the slides: Lecture16.pdf

Q-Q plots, moment testing, and normality testing in R:
NBS Weight Data.csv
, qqplot moment normality demo.R

Wednesday, September 14

Homework #3 (due one week from today): HW3_Moments_Outliers.pdf

Lecture 17: Testing for Outliers, part 1 (26 min) - hardcopy of the slides: Lecture17.pdf

Lecture 18: Testing for Outliers, part 2 (18 min) - hardcopy of the slides: Lecture18.pdf

Reading: Procedures for Detecting Outlying Observations in Samples_Grubbs_1969.pdf

Tables of critical values for the Grubbs' Outlier Tests: Grubbs Test Critical Values.pdf

Friday, September 16

Lecture 19: Final Thoughts on Outliers (29 min) - hardcopy of the slides: Lecture19.pdf

Lecture 20: Performing Outlier Tests in Excel and R: Lecture20.xlsx, Outliers demo.R

Monday, September 19

Lecture 21: Leverage in Regression (22 min) - hardcopy of the slides: Lecture21.pdf

Lecture 22: Influence in Regression (20 min) - hardcopy of the slides: Lecture22.pdf

Lecture 23: Leverage and Influence in Excel and R (17 min):
Flow Rate Calibration.csv
, Influence demo.R, Lecture23.xlsx

Wednesday, September 21

Homework #4 (due in one week): HW4_Influence_Scedasticity.pdf

Lecture 24: Heteroscedasticity: When Variance Varies (23 min) - hardcopy of the slides: Lecture24.pdf

Lecture 25: Testing for Homoscedasticity in Excel and R (15 min): Lecture25.xlsx, Heteroscedasticity.R

Friday, September 23

Lecture 26: Correcting for Heteroscedasticity (16 min) - hardcopy of the slides: Lecture26.pdf

Lecture 27: Data Transformations in R (19 min): OLS and four Graphs.R, Box-Cox demo.R

Monday, September 26

Lecture 28: Weighted Regression (18 min) - hardcopy of the slides: Lecture28.pdf

Lecture 29: Weighted Regression in R (8 min) - Weighted regression demo.R

Excel Resource - the Real-Statistics Website: www.real-statistics.com

Wednesday, September 28

Lecture 30: Total Regression, part 1 (19 min) - hardcopy of the slides: Lecture30.pdf

Lecture 31: Total Regression, part 2 (22 min) - hardcopy of the slides: Lecture31.pdf

Friday, September 30

Homework #5 (due in one week): HW5_Weighted_Total_regression.pdf

Lecture 32: Total Regression, part 3 (18 min) - hardcopy of the slides: Lecture32.pdf

Lecture 33: Total Regression in R (19 min):
Total Regression_effective variance.R, Deming Regression.R

Monday, October 3

Lecture 34: The Wrong Model (22 min) - hardcopy of the slides: Lecture34.pdf

Wednesday, October 5

Lecture 35: The Wrong Model, part 2 (27 min) - hardcopy of the slides: Lecture35.pdf

Lecture 36: Goodness of Fit tests in R (11 min) - Regression Goodness of Fit.R

Friday, October 7

Homework #6 (due in one week): HW6_Autoregression.pdf

Lecture 37: Independence of Residuals (27 min) - hardcopy of the slides: Lecture37.pdf

Lecture 38: Residual Independence in Excel and R (18 min) - Lecture38.xlsx, Lag plot and Runs test.R

Monday, October 10

Lecture 39: Autocorrelation in Time Series (32 min) - hardcopy of the slides: Lecture39.pdf

Lecture 40: Time Series Autocorrelation in Excel and R (24 min):
Lecture40.xlsx, Durbin-Watson test.R

Tables of critical values for the Durbin-Watson Test: Durbin_Watson_tables.pdf

More Durbin-Watson critical values can be found here: web.stanford.edu/~clint/bench/dwcrit.htm

Wednesday, October 12

Lecture 41: Regression Review (17 min) - hardcopy of the slides: Lecture41.pdf

Friday, October 14

Review materials for Exam #1: Exam1_Review.pdf

Exam #1 Take-Home piece given out

Monday, October 17

Exam #1 (In-Class piece)

Wednesday, October 19

Homework #7 (due in one week): HW7_Multiple_Regression.pdf

Lecture 42: Multiple Regression (23 min) - hardcopy of the slides: Lecture42.pdf

Lecture 43: Comparing Models (17 min) - hardcopy of the slides: Lecture43.pdf

Lecture 44: Multiple Regression in Excel and R (21 min):
Lecture44.xlsx, Multiple Regression.R

Friday, October 21

Review of Exam #1 solution

Writing Project (Due Nov. 18): Data to Decisions Writing Project.pdf

Monday, October 24

Lecture 45: Best Subset Regression (18 min) - hardcopy of the slides: Lecture45.pdf

Lecture 46: Best Subset Regression in R (19 min) - Best Subset Model.R

Wednesday, October 26

Homework #8 (due in one week): HW8_Multicollinearity.pdf
Paper explaining the data set for HW#8, click here.

Lecture 47: Multicollinearity (18 min) - hardcopy of the slides: Lecture47.pdf

Lecture 48: Standardized Variables (10 min) - hardcopy of the slides: Lecture48.pdf

Lecture 49: Multicollinearity in Excel and R (9 min): Lecture49.xlsx, Standardized Regression.R

Friday, October 28

Lecture 50: Detecting Multicollinearity (18 min) - hardcopy of the slides: Lecture50.pdf

Lecture 51: Addressing Multicollinearity (14 min) - hardcopy of the slides: Lecture51.pdf

Lecture 52: Detecting Multicollinearity and Ridge Regression in R (24 min):
Detecting Multicollinearity.R, Ridge Regression.R

Monday, October 31

Lecture 53: Principal Component Analysis (25 min) - hardcopy of the slides: Lecture53.pdf

Lecture 54: Principal Component Analysis in R (16 min) - Principal Components.R

Wednesday, November 2

Lecture 55: Robust Estimation (20 min) - hardcopy of the slides: Lecture55.pdf

Lecture 56: Robust Regression (22 min) - hardcopy of the slides: Lecture56.pdf

Lecture 57: Robust Regression in R (11 min) - Robust Regression.R

Friday, November 4

Homework #9 (due in one week): HW9_Logistic_Regression.pdf; HW9.xlsx

Lecture 58: Generalized Linear Modeling (19 min) - hardcopy of the slides: Lecture58.pdf

Lecture 59: Other Regression Topics (17 min) - hardcopy of the slides: Lecture59.pdf

Lecture 60: Generalized Linear Modeling in R (21 min): Generalized Linear Modeling.R

Monday, November 7

Lecture 61: Logistic Modeling Example -The sinking of the Titanic (40 min): Logistic Regression.R

Wednesday, November 9

Lecture 62: Model Building (26 min) - hardcopy of the slides: Lecture62.pdf

Lecture 63: Model Building in R (13 min): Model building.R

Model Building Contest: Model Building Contest.pdf, BlackFridayTrain.csv

Model Building Contest entries due Nov. 30 in class.

Friday, November 11

Homework #10 (due in one week): HW10_DOE.pdf

Lecture 64: Introduction to Design of Experiments (26 min) - hardcopy of the slides: Lecture64.pdf

Lecture 65: Regression Design (21 min) - hardcopy of the slides: Lecture65.pdf

Lecture 66: Simple Regression Design in R (11 min): DOE simple example.R, DOE Simple Example.xlsx

Monday, November 14

Lecture 67: Blocking in Experimental Design (21 min) - hardcopy of the slides: Lecture67.pdf

Lecture 68: Factorial Design of Experiments (30 min) - hardcopy of the slides: Lecture68.pdf

Lecture 69: Analysis of Covariance in R (13 min): DOE analysis of covariance.R
DOE Analysis of Covariance.xlsx

Lecture 70: Factorial Design in R (30 min): DOE Factorial Design.R, FactorialDesign.csv
Factorial Designs.xlsx

Wednesday, November 16

Lecture 71: Response Surface Modeling (20 min) - hardcopy of the slides: Lecture71.pdf

Lecture 72: Final Thoughts on Design of Experiments (18 min) - hardcopy of the slides: Lecture72.pdf

Lecture 73: Response Surface Modeling in R (14 min): DOE Response Surface.R

Friday, November 18

Review materials for Exam #2: Exam2_Review.pdf

Exam #2 Take-Home Piece given out

Monday, November 21

Exam #2 (In-Class Piece)

Wednesday, November 23

No class - Happy Thanksgiving!

Friday, November 25

No class - Happy Thanksgiving!

Monday, November 28

Lecture 74: Bayesian Regression, part 1 (26 min) - hardcopy of the slides: Lecture74.pdf

Lecture 75: Bayesian Regression, part 2 (24 min) - hardcopy of the slides: Lecture75.pdf

Wednesday, November 30

Model Building Contest Results!

Friday, December 2

Lecture 76: Bayesian Regression, part 3 (xx min) - hardcopy of the slides: Lecture76.pdf

Lecture 77: Bayesian Regression in R (xx min):

Monday, December 5

 

 

 

 

The items below are under construction...

 

Wednesday, December 2

Lecture 29: Measurement Uncertainty - hardcopy of the slides: Lecture29.pdf

NIST Technical Note 1297 (1994): Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results

GUM: Guide to the Expression of Uncertainty in Measurement

Friday, December 4

Last Day of Class!

Lecture 30: Propagation of Uncertainty - hardcopy of the slides: Lecture30.pdf