PPOL 502-07: Reg. Methods for Policy Analysis

class: center, middle, inverse, title-slide

# PPOL 502-07: Reg. Methods for Policy Analysis
## Week 12: Predictive Models
### Alexander Podkul, PhD
### Spring 2022

---

## Today's Class Outline

* Course Schedule
* Predictions (Review)
* Prediction Errors 
* Using Predictions
* Working with Out of Sample Data
* Fitting the Model
* Where We're Going Next 
* __Break__
* Working in Stata

---
## Course Schedule

__Tonight (4/13)__

* {Nothing due}

__Next week (4/20)__

* Data Project due

__The final week (4/27)__

* {Nothing due}
* Post suggested readings! (or email them to me)

__The final exam (5/6)__

* Final exam!

---
## Data Project Comments

General update:

* (Unless you have an extenuating circumstance) Feedback has been distributed!
* Projects seem to be in a good place!

--
A few focus areas as we approach the final stretch:

* Consider multiple specifications (telling the whole story)
* Think hard about omitted variable bias 
* Perform diagnostics 
* Transforming variables may add to your story 
* It's never too late to consider interaction or quadratic models 😄

---
## Reviewing Predictions

So far this semester, we've already discussed a number of prediction-related topics:

* Fitted Values 
* Adapting Predictions for `$log(y)$`
* Standard Errors of Fitted Values

---
### Fitted Values

(Review) Already in this course we've discussed making predictions from our models and our data. Early in the semester we called them 'fitted values' and later we used them in thinking about interpreting interaction models.

If we estimated a model such that: 
`$$\hat{y} = 11 + 2x_1 + 4x_2 + 0.25x_3$$`

--
we could find the following _fitted values_ (also known as _predictions_) from our data.

---
### Adapting Predictions for `$log(y)$`

`$$log(y) = \beta_0 + \beta_1x_1 + \beta_2x_2$$`

(Review) Due to our errors being distributed according to a log-normal distribution, predictions from a log-level or log-log model are going to under-estimate the expected value of `$y$` when we convert our `$\hat{log(y)}$` to `$y$`.

--
<img src="Week12_files/figure-html/unnamed-chunk-2-1.png" style="display: block; margin: auto;" />

---
### Adapting Predictions for `$log(y)$`

In other words, the expected method  
`$$E(y|X) = e^{\hat{log(y)}}$$`
would create underestimates.

--
To fix this, we add a correction factor such that: 
`$$E(y|X) = \hat{\alpha}_0e^{\hat{log(y)}}$$`

where we estimate `$\hat{\alpha}$` by looking at the average exponentiated residual or `$\check{\alpha}$` by standardizing the exponentiated values by the raw values of `$y$`.

---
### Standard Error of the Prediction

(Review) Estimating a prediction is easy (plug it in the equation!). Estimating the standard error is a bit more complicated.

--
If we have an estimated equation such that:
`$$\hat{y} = \hat{\beta_0} + \hat{\beta_1}x_1 + \hat{\beta_2}x_2 + ... +  \hat{\beta_k}x_k$$`

--
then the parameter we aim to estimate is: 
`$$\theta_0 = \beta_0 + \beta_1c_1 + \beta_2c_2 + ... +  \beta_kc_k$$`

where `$c$` represents particular values (or constants) for each of `$k$` independent variables.

--
Finally, our estimator of `$\theta_0$` becomes: 
`$$\hat{\theta_0} = \hat{\beta_0} + \hat{\beta_1}c_1 + \hat{\beta_2}c_2 + ... +  \hat{\beta_k}c_k$$`

---
### Standard Error of the Prediction
To obtain a standard error for `$\hat{\theta_0}$` , we need to consider the
linear combination of our OLS estimators (because the value is dependent on all values of `$\hat{\beta_j}$`, unless `$c_j$` = 0).

--
To find the standard error associated with the _expected value_ of `$y$`, we can:

1. re-arrange our parameter equation so that
`$$\theta_0 = \beta_0 + \beta_1c_1 + \beta_2c_2 + ... +  \beta_kc_k$$`
`$$\beta_0 = \theta_0  - \beta_1c_1 - \beta_2c_2 - ... -  \beta_kc_k$$`

2. substitute our rearranged parameter equation into our regression formula so that

`$$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... +  \beta_kx_k$$`

`$$y = \theta_0 + \beta_1(x_1 - c_1) + \beta_2(x_2 - c_2) + ... +  \beta_k(x_k - c_k)$$`

--
... which is the equivalent to estimating a regression for `$y$` on `$x_1 - c_1$`, `$x_2 - c_2$` and using the standard error from the intercept term!

---
### Prediction Examples
<table class="texreg" style="margin: 10px auto;border-collapse: collapse;border-spacing: 0px;caption-side: bottom;color: #000000;border-top: 2px solid #000000;">
<caption>Statistical models</caption>
<thead>
<tr>
<th style="padding-left: 5px;padding-right: 5px;">&nbsp;</th>
<th style="padding-left: 5px;padding-right: 5px;">Clinton Share</th>
<th style="padding-left: 5px;padding-right: 5px;">Clinton Share</th>
</tr>
</thead>
<tbody>
<tr style="border-top: 1px solid #000000;">
<td style="padding-left: 5px;padding-right: 5px;">Intercept</td>
<td style="padding-left: 5px;padding-right: 5px;">54.196<sup>***</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">43.807<sup>***</sup></td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(2.014)</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.413)</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Median Age</td>
<td style="padding-left: 5px;padding-right: 5px;">-0.106<sup>*</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.054)</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Per Capita Income</td>
<td style="padding-left: 5px;padding-right: 5px;">-0.001<sup>***</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.000)</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Median Age - 35</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">-0.106<sup>*</sup></td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.054)</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Per Capita Income - 10000</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">-0.001<sup>***</sup></td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.000)</td>
</tr>
<tr style="border-top: 1px solid #000000;">
<td style="padding-left: 5px;padding-right: 5px;">R<sup>2</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">0.052</td>
<td style="padding-left: 5px;padding-right: 5px;">0.052</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Adj. R<sup>2</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">0.051</td>
<td style="padding-left: 5px;padding-right: 5px;">0.051</td>
</tr>
<tr style="border-bottom: 2px solid #000000;">
<td style="padding-left: 5px;padding-right: 5px;">Num. obs.</td>
<td style="padding-left: 5px;padding-right: 5px;">2704</td>
<td style="padding-left: 5px;padding-right: 5px;">2704</td>
</tr>
</tbody>
<tfoot>
<tr>
<td style="font-size: 0.8em;" colspan="3"><sup>***</sup>p &lt; 0.001; <sup>**</sup>p &lt; 0.01; <sup>*</sup>p &lt; 0.05</td>
</tr>
</tfoot>
</table>

... which provides the prediction (and significance) for when median age = 35 and per capita income = $10,000 (i.e. the predictions of the expected value of our dependent variable given these covariates!)

---
## Prediction Interval

The previous slides are useful by producing the standard error of `$\hat{y}$`.

However, when wrestling with predictions, we might be interested not in the standard error of the _expectation_ but rather the standard error of our _predictions_.

---
## Prediction Interval 
<img src="Week12_files/figure-html/unnamed-chunk-5-1.png" style="display: block; margin: auto;" />

---
## Prediction Interval 
<img src="Week12_files/figure-html/unnamed-chunk-6-1.png" style="display: block; margin: auto;" />

---
## Prediction Interval 
<img src="Week12_files/figure-html/unnamed-chunk-7-1.png" style="display: block; margin: auto;" />

---
## Prediction Interval 
<img src="Week12_files/figure-html/unnamed-chunk-8-1.png" style="display: block; margin: auto;" />

---
## Prediction Interval

Generally speaking, the __confidence interval__ refers to the range of _expected_ values. A __prediction interval__ refers to the fuller uncertainty associated with a single value.

--
Assume we have two intervals, both at 95%: 
- the confidence interval band reveals the 95% confidence of the interval holding the expected value (think about sampling error) 
- the prediction interval reveals the 95% confidence of the interval containing various predictions (think sampling error + population error)

--
In other words, the prediction interval accounts for additional variance in unobserved error

---
### Calculating the Prediction Interval

As mentioned on the previous slide, our prediction interval is the sum of sampling error and population error, or:
`$$Var(\hat{e^0}) = Var(\hat{y^0}) + Var(u^0) = Var(\hat{y^0}) + \sigma^2$$`

--
Expressed slightly differently, we tend to estimate a standard error of the prediction error, or: 
`$$se(\hat{e}^0) = \sqrt{se(\hat{y^{0}})^2 + \hat{\sigma^2}}$$`

where:

* `$se(\hat{y^{0}})$` is the estimated standard error of our prediction `$\hat{y^0}$`}
* `$\hat{\sigma^2}$` is the variance of the residuals

--
which we often express as an interval estimated in the familiar way as: 
`$$\hat{y^0} \pm t * se(\hat{e}^0)$$`

---
### Prediction Interval Example

<table class="texreg" style="margin: 10px auto;border-collapse: collapse;border-spacing: 0px;caption-side: bottom;color: #000000;border-top: 2px solid #000000;">
<caption>Statistical models</caption>
<thead>
<tr>
<th style="padding-left: 5px;padding-right: 5px;">&nbsp;</th>
<th style="padding-left: 5px;padding-right: 5px;">Clinton Share</th>
<th style="padding-left: 5px;padding-right: 5px;">Clinton Share</th>
</tr>
</thead>
<tbody>
<tr style="border-top: 1px solid #000000;">
<td style="padding-left: 5px;padding-right: 5px;">Intercept</td>
<td style="padding-left: 5px;padding-right: 5px;">54.196<sup>***</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">43.807<sup>***</sup></td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(2.014)</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.413)</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Median Age</td>
<td style="padding-left: 5px;padding-right: 5px;">-0.106<sup>*</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.054)</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Per Capita Income</td>
<td style="padding-left: 5px;padding-right: 5px;">-0.001<sup>***</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.000)</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Median Age - 35</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">-0.106<sup>*</sup></td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.054)</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Per Capita Income - 10000</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">-0.001<sup>***</sup></td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.000)</td>
</tr>
<tr style="border-top: 1px solid #000000;">
<td style="padding-left: 5px;padding-right: 5px;">R<sup>2</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">0.052</td>
<td style="padding-left: 5px;padding-right: 5px;">0.052</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Adj. R<sup>2</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">0.051</td>
<td style="padding-left: 5px;padding-right: 5px;">0.051</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Num. obs.</td>
<td style="padding-left: 5px;padding-right: 5px;">2704</td>
<td style="padding-left: 5px;padding-right: 5px;">2704</td>
</tr>
<tr style="border-bottom: 2px solid #000000;">
<td style="padding-left: 5px;padding-right: 5px;">RMSE</td>
<td style="padding-left: 5px;padding-right: 5px;">9.918</td>
<td style="padding-left: 5px;padding-right: 5px;">9.918</td>
</tr>
</tbody>
<tfoot>
<tr>
<td style="font-size: 0.8em;" colspan="3"><sup>***</sup>p &lt; 0.001; <sup>**</sup>p &lt; 0.01; <sup>*</sup>p &lt; 0.05</td>
</tr>
</tfoot>
</table>

---
### Prediction Interval Example

If we want to find the prediction interval where median age = 35 and per capita income is 10000, we can solve using:

`$$se(\hat{e}^0) = \sqrt{se(\hat{y^{0}})^2 + \hat{\sigma^2}}$$`
`$$se(\hat{y^{0}}) = 0.413$$`

`$$\hat{\sigma} = 9.918$$`

`$$se(\hat{e}^0) = \sqrt{0.413^2 + 9.918^2}$$`

`$$se(\hat{e}^0) = 9.926$$`

---
### Prediction Interval Example
<img src="Week12_files/figure-html/unnamed-chunk-10-1.png" style="display: block; margin: auto;" />

---
### Prediction Interval Example
<img src="Week12_files/figure-html/unnamed-chunk-11-1.png" style="display: block; margin: auto;" />

---
## Using Predictions

To take a step back, there is often a distinction between __explanatory modeling__ and __predictive modeling__ (Schmueli 2020).

* Explanatory modeling - the use of statistical models for testing causal explanations, where we can make inferences to test hypotheses
* Predictive modeling - applying a statistical model or algorithm to data for the purpose of predicting new (or future) observations

--
A useful framework for distinguishing between the two is identifying the focus of the following axes:

* _causation-association_ - explanatory seeks the causal; predictive identifies the association

--
* _theory-data_ - theory often dictates explanatory models; data often drives predictive models

--
* _retrospective-prospective_ - explanatory is backward looking; predictive is forward looking

--
* _bias-variance_ - explanatory seeks to minimize bias; predictive to minimize variance

---
## Using Predictions

`$$y = \beta_0 + \beta_1x_1 + \beta_2x_2$$`

`$$\hat{y} = \hat{\beta_0} + \hat{\beta_1}x_1 + \hat{\beta_2}x_2$$`

Using the same model, we might focus on different elements:

* explanatory would likely focus on the estimates for `$\beta$`
* predictive would likely focus on how well we can estimate the fitted values

---
## Working with Out of Sample Data

So far in class we've mostly just spoken about our "data" and we've used that data to estimate various models for explanatory purposes. But as we pivot from explanatory to predictive we also need to think of our data differently.

One way to think of our data is to reference "in-sample" vs. "out-of-sample" data. Generally this means refers to whether or not we collected and used a particular datum to estimate our model.

---
## Working with Out of Sample Data: Example 1

Imagine we estimate the following equation:

`$$ViolentC = \beta_0 + \beta_1PoliceFunding + \beta_2RegB + \beta_2RegC + \beta_2RegD + \beta_2RegE$$`

---
## Working with Out of Sample Data: Example 1

Out of sample data might help us answer...

_What if State 41 in Region E increased its annual police funding by five points?_

Current value: 3545

Current fitted value: 1708

New fitted value: 1793

Confidence interval: [1284, 2301]

Prediction interval: [778, 2807]

--
_What if a new state were added to region B and had the maximum annual police funding?_

Current value: NA

Current fitted value: NA

New fitted value: 1181

Confidence interval: [547, 1816]

Prediction interval: [98, 2265]

---
## Working with Out of Sample Data: Example 2

To explore a second example, now imagine using our admissions data we have estimated the model: 
`$$P(Admitted = 1) = \Phi(\beta_0 + \beta_1GRE + \beta_2CGPA + \beta_3Research)$$`

---
## Working with Out of Sample Data: Example 2

_What does the model predict for a student with a GRE score of 330, a GPA of 8, and no research experience?_

Probability: 44%

---
## Fitting the Model

We may be tempted when building a model to have the _best_ fit possible (think: highest `$R^2$` or lowest `$RMSE$`, however that might not always be the smartest decision.

In part, because we might accidentally come up with the perfect model for our _sample_ but not for our _population_.

---
## Fitting the Model

Let's take an example where we have 100 cases in our population. We _know_ the relationship is: 
`$$y = 1 + .4x + x^2 + u$$`

but let's say we sample 40 cases and try to estimate the full relationship. We might consider the following models:

--
<table class="texreg" style="margin: 10px auto;border-collapse: collapse;border-spacing: 0px;caption-side: bottom;color: #000000;border-top: 2px solid #000000;">
<caption>Statistical models</caption>
<thead>
<tr>
<th style="padding-left: 5px;padding-right: 5px;">&nbsp;</th>
<th style="padding-left: 5px;padding-right: 5px;">Model 1</th>
<th style="padding-left: 5px;padding-right: 5px;">Model 2</th>
<th style="padding-left: 5px;padding-right: 5px;">Model 3</th>
<th style="padding-left: 5px;padding-right: 5px;">Model 4</th>
</tr>
</thead>
<tbody>
<tr style="border-top: 1px solid #000000;">
<td style="padding-left: 5px;padding-right: 5px;">Intercept</td>
<td style="padding-left: 5px;padding-right: 5px;">-15.287<sup>*</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">-4.625</td>
<td style="padding-left: 5px;padding-right: 5px;">-2.101</td>
<td style="padding-left: 5px;padding-right: 5px;">-2.627</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(6.414)</td>
<td style="padding-left: 5px;padding-right: 5px;">(4.356)</td>
<td style="padding-left: 5px;padding-right: 5px;">(4.524)</td>
<td style="padding-left: 5px;padding-right: 5px;">(5.662)</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">X</td>
<td style="padding-left: 5px;padding-right: 5px;">11.024<sup>***</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">1.204</td>
<td style="padding-left: 5px;padding-right: 5px;">3.314</td>
<td style="padding-left: 5px;padding-right: 5px;">3.339</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.988)</td>
<td style="padding-left: 5px;padding-right: 5px;">(1.464)</td>
<td style="padding-left: 5px;padding-right: 5px;">(1.918)</td>
<td style="padding-left: 5px;padding-right: 5px;">(1.951)</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">X^2</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">0.994<sup>***</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">0.286</td>
<td style="padding-left: 5px;padding-right: 5px;">0.390</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.134)</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.449)</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.800)</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">X^3</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">0.045</td>
<td style="padding-left: 5px;padding-right: 5px;">0.025</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.027)</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.130)</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">X^4</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">0.001</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">&nbsp;</td>
<td style="padding-left: 5px;padding-right: 5px;">(0.006)</td>
</tr>
<tr style="border-top: 1px solid #000000;">
<td style="padding-left: 5px;padding-right: 5px;">R<sup>2</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">0.766</td>
<td style="padding-left: 5px;padding-right: 5px;">0.906</td>
<td style="padding-left: 5px;padding-right: 5px;">0.913</td>
<td style="padding-left: 5px;padding-right: 5px;">0.913</td>
</tr>
<tr>
<td style="padding-left: 5px;padding-right: 5px;">Adj. R<sup>2</sup></td>
<td style="padding-left: 5px;padding-right: 5px;">0.760</td>
<td style="padding-left: 5px;padding-right: 5px;">0.901</td>
<td style="padding-left: 5px;padding-right: 5px;">0.906</td>
<td style="padding-left: 5px;padding-right: 5px;">0.903</td>
</tr>
<tr style="border-bottom: 2px solid #000000;">
<td style="padding-left: 5px;padding-right: 5px;">Num. obs.</td>
<td style="padding-left: 5px;padding-right: 5px;">40</td>
<td style="padding-left: 5px;padding-right: 5px;">40</td>
<td style="padding-left: 5px;padding-right: 5px;">40</td>
<td style="padding-left: 5px;padding-right: 5px;">40</td>
</tr>
</tbody>
<tfoot>
<tr>
<td style="font-size: 0.8em;" colspan="5"><sup>***</sup>p &lt; 0.001; <sup>**</sup>p &lt; 0.01; <sup>*</sup>p &lt; 0.05</td>
</tr>
</tfoot>
</table>

---
## Fitting the Model

We might assume that model four is the best model for making predictions considering the high `$R^2$` and adjusted `$R^2$`.

--
But let's consider which model performs best on a _different_ random sample from our population. Here, we'll sample 20 new observations and consider the RMSE. We can compare the RMSE from this out-of-sample group to the in-sample.

--
<table class=" lightable-paper" style='font-family: "Arial Narrow", arial, helvetica, sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:right;"> RMSE Mod 1 </th>
   <th style="text-align:right;"> RMSE Mod 2 </th>
   <th style="text-align:right;"> RMSE Mod 3 </th>
   <th style="text-align:right;"> RMSE Mod 4 </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Original 40 </td>
   <td style="text-align:right;"> 15.10489 </td>
   <td style="text-align:right;"> 11.4661 </td>
   <td style="text-align:right;"> 11.45303 </td>
   <td style="text-align:right;"> 11.08218 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> New 20 </td>
   <td style="text-align:right;"> 35.18467 </td>
   <td style="text-align:right;"> 13.4988 </td>
   <td style="text-align:right;"> 12.94296 </td>
   <td style="text-align:right;"> 57.99199 </td>
  </tr>
</tbody>
</table>

---
### Overfitting and Underfitting

.pull-left[
This example introduces us to:
- underfitting: not great fit to the training data, not great fit to other data (high bias)
- overfitting: great fit to our training data, not as great generalization to other data (such as the testing set) (high variance)
]
.pull-right[
<img src="Week12_files/figure-html/unnamed-chunk-16-1.png" width="90%" />
]

---
### Overfitting and Underfitting

.pull-left[
This example introduces us to:
- __underfitting__: not great fit to the training data, not great fit to other data (high bias)
- overfitting: great fit to our training data, not as great generalization to other data (such as the testing set) (high variance)
]
.pull-right[
<img src="Week12_files/figure-html/unnamed-chunk-17-1.png" width="90%" />
]

---
### Overfitting and Underfitting

.pull-left[
This example introduces us to:
- underfitting: not great fit to the training data, not great fit to other data (high bias)
- __overfitting__: great fit to our training data, not as great generalization to other data (such as the testing set) (high variance)
]
.pull-right[
<img src="Week12_files/figure-html/unnamed-chunk-18-1.png" width="90%" />
]

---
### Overfitting and Underfitting

---
### Prediction Workflow

The workflow for creating predictive models is _slightly_ different than how we've discussed in the past -- in part because we are now quite sensitive to the accuracy of out-of-sample predictions.

--
To prevent overfitting, we will often follow:

* Randomly dividing our data into training data and testing data 
* Estimate a model using our training data (and _only_ our training data)
* Use this model (estimated by the training data) to make predictions on the testing data 
* Assess the accuracy of our model by comparing the predictions with the actual observed values.

--
And this workflow works for OLS, logit, probit, and many others.

--
_Note_: This is a bit different than how we think of the same process for explanatory models.

* We're _willingly_ using less data to estimate our model (gasp!)
* We're no longer just using in-sample or hypothetical data points to render predictions.

---
### Prediction Workflow Example

Let's say we have a data set of the GSS and we're interested in developing a simple model to predict whether someone voted (in the 2008 election).

<div id="htmlwidget-ba1db11e62e7dc98af18" style="width:100%;height:auto;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-ba1db11e62e7dc98af18">{"x":{"filter":"none","vertical":false,"fillContainer":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20"],[2012,2012,2012,2012,2012,2012,2012,2012,2012,2012,2012,2012,2012,2012,2012,2012,2012,2012,2012,2012],[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20],["WORKING PART TIME","WORKING PART TIME","WORKING FULL TIME",null,"RETIRED",null,"KEEPING HOUSE","KEEPING HOUSE","KEEPING HOUSE","WORKING FULL TIME",null,"WORKING PART TIME","KEEPING HOUSE",null,"WORKING PART TIME","WORKING FULL TIME","WORKING FULL TIME","WORKING FULL TIME","WORKING FULL TIME","SCHOOL"],["SOMEONE ELSE","SOMEONE ELSE","SOMEONE ELSE","SOMEONE ELSE","SOMEONE ELSE","SOMEONE ELSE","SOMEONE ELSE",null,null,"SOMEONE ELSE","SOMEONE ELSE",null,null,"SOMEONE ELSE","SOMEONE ELSE","SOMEONE ELSE","SOMEONE ELSE","SOMEONE ELSE","SOMEONE ELSE",null],["PRIVATE","PRIVATE","PRIVATE","PRIVATE","GOVERNMENT","PRIVATE","PRIVATE",null,null,"PRIVATE","PRIVATE","PRIVATE",null,"GOVERNMENT","PRIVATE","PRIVATE","PRIVATE","PRIVATE","PRIVATE",null],["Never married","Never married","Married","Married","Separated","Widowed","Married","Separated","Married","Never married","Divorced","Never married","Never married","Divorced","Never married","Never married","Married","Divorced","Married","Never married"],[1,2,1,2,0,4,2,2,2,0,4,2,6,7,6,2,5,2,9,3],[0,0,2,2,3,2,2,3,2,0,6,3,4,6,4,4,3,5,6,0],[22,21,42,49,70,50,35,24,28,28,55,36,28,59,52,35,36,47,55,18],["4 years","12th grade","12th grade","1 yr coll","4 years","7 years","3 years","11th grade","9th grade","5 years","10th grade","4 years","12th grade","12th grade","4th grade","1 yr coll","12th grade","1 yr coll","12th grade","12th grade"],["Bachelor deg","HS","HS","HS","Bachelor deg","Bachelor deg","Junior Coll","<HS","<HS","Bachelor deg","<HS","Bachelor deg","HS","HS","<HS","HS","HS","HS","HS","HS"],["Male","Male","Male","Female","Female","Female","Female","Female","Female","Female","Male","Female","Female","Female","Male","Female","Male","Female","Male","Female"],["White","White","Other","White","Black","White","White","Other","Black","White","White","Black","White","Black","Other","Black","White","Black","Other","Other"],["Moderate","SlghtCons","SlghtCons","SlghtCons","Liberal","Moderate","Moderate","Moderate","Conserv","Liberal","ExtrmLib","Liberal","Moderate",null,"ExtrmCons","SlghtLib","Moderate","Conserv","Moderate","Moderate"],["IndDem","IndRep","IndDem","WkRep","StrDem","StrDem","Ind","Ind","Ind","StrDem","StrDem","WkDem","Ind","Ind","Ind","WkRep","WkRep","IndDem",null,"Ind"],["SAME ST, DIF CITY","SAME CITY","DIFFERENT STATE","SAME ST, DIF CITY","SAME CITY","SAME ST, DIF CITY","SAME CITY","SAME CITY",null,"SAME CITY","SAME CITY","DIFFERENT STATE","DIFFERENT STATE","SAME CITY","DIFFERENT STATE","SAME ST, DIF CITY","DIFFERENT STATE","SAME CITY","DIFFERENT STATE","SAME ST, DIF CITY"],["YES","YES","YES","YES","YES","YES","YES","NO","YES","YES","YES","YES","YES","YES","NO","YES","NO","YES","NO","YES"],["$150000 OR OVER","$150000 OR OVER","$110000 TO $129999","$130000 TO $149999","$50000 TO 59999","$25000 TO 29999","$30000 TO 34999","$5 000 TO 5 999","UNDER $1 000","$30000 TO 34999","UNDER $1 000","$50000 TO 59999","$8 000 TO 9 999","$22500 TO 24999",null,"$50000 TO 59999","$50000 TO 59999","$40000 TO 49999","$60000 TO 74999","$22500 TO 24999"],[null,null,"$110000 TO $129999",null,null,null,null,null,null,"$30000 TO 34999",null,"$6 000 TO 6 999",null,null,"$15000 TO 17499","$50000 TO 59999","$50000 TO 59999","$40000 TO 49999","$60000 TO 74999",null],["NEW ENGLAND","NEW ENGLAND","NEW ENGLAND","NEW ENGLAND","MIDDLE ATLANTIC","MIDDLE ATLANTIC","MIDDLE ATLANTIC","MIDDLE ATLANTIC","MIDDLE ATLANTIC","MIDDLE ATLANTIC","NEW ENGLAND","NEW ENGLAND","NEW ENGLAND","NEW ENGLAND","NEW ENGLAND","NEW ENGLAND","NEW ENGLAND","MIDDLE ATLANTIC","MIDDLE ATLANTIC","MIDDLE ATLANTIC"],[14,14,14,14,24,24,24,24,24,24,56,56,56,56,56,56,56,125,125,125],["Did not vote",null,"Voted","Voted","Voted","Voted","Did not vote",null,"Voted","Voted","Voted","Did not vote","Did not vote","Voted","Did not vote","Voted",null,"Voted","Did not vote",null],[null,null,"Obama","McCain","Obama","Obama",null,null,"Obama","Obama","McCain",null,null,"Obama",null,"Obama",null,"Obama",null,null],[null,null,"About right",null,null,"About right","Too little",null,"About right","About right",null,"Too much",null,"Too much",null,"About right",null,"Too much",null,null],[null,null,"About right",null,null,"Too little","Too little",null,"Too little","Too little",null,"Too little",null,"Too little",null,"Too little",null,"Too little",null,null],[null,null,"Too much",null,null,"Too little","Too little",null,"About right","Too little",null,"Too little",null,"Too little",null,"Too little",null,"Too little",null,null],[null,null,"About right",null,null,"Too little","Too little",null,"Too little",null,null,"Too little",null,"Too little",null,"Too little",null,"About right",null,null],[null,null,"About right",null,null,"About right","Too little",null,"About right","About right",null,"Too little",null,"Too little",null,"Too little",null,"Too little",null,null],[null,null,null,null,null,"Too little","About right",null,"About right","Too little",null,"Too little",null,"Too much",null,"Too little",null,"About right",null,null],[null,null,"Too little",null,null,"About right","Too little",null,"Too little","Too little",null,"Too little",null,"Too little",null,"About right",null,"Too little",null,null],[null,null,"About right",null,null,"Too much","About right",null,"Too little","Too little",null,"Too little",null,"Too little",null,"Too little",null,"About right",null,null],[null,null,"Too much",null,null,"Too much","Too little",null,"About right","About right",null,"About right",null,"Too much",null,"Too little",null,"Too much",null,null],[null,null,"Too much",null,null,"Too much","About right",null,"About right","About right",null,"Too much",null,"Too much",null,"Too little",null,"Too much",null,null],[null,null,"About right",null,null,"Too much","Too much",null,"Too little","Too little",null,"Too little",null,"Too little",null,"About right",null,"Too much",null,null],["Too much","About right","Too little","Too little","Too little","Too little","Too little","About right","About right","Too little","Too little","Too little","Too much","Too much","About right","About right","Too little","Too little","About right","Too little"],["Too little","Too much","About right",null,"Too little","Too much","Too little","Too much","About right","Too little","Too little","Too little","About right","Too little","Too little","Too little","Too little","Too little","Too little","Too little"],["Too little","About right",null,"About right","About right","About right","About right","About right","Too little","Too little","Too little","About right","About right","About right","About right","About right","About right","About right","About right","Too little"],["About right","About right","Too little","Too much","About right","Too little","About right","About right","Too little",null,"Too little","Too little","Too little","Too little","Too little","About right","Too little","About right","About right","About right"],["About right","About right","Too little",null,"About right","Too little","Too little","Too little","Too little","Too little","Too little","Too little","About right","Too little","About right","Too little","About right","Too little","Too little",null],["About right","Too little","Too little","Too much","Too little","Too little","About right","About right","Too much","About right",null,"About right","Too much","About right","Too little","About right",null,"Too much","Too little","About right"],["Too little","Too little","About right","Too little","Too little","Too little","About right","Too little","About right","Too little","Too little","Too little","About right","Too little","About right","About right","Too little","Too little","Too little","Too little"],[null,3,5,7,7,null,2,1,null,1,null,1,4,null,7,1,4,1,7,null],[null,"ALLOWED","ALLOWED",null,null,"ALLOWED",null,null,"ALLOWED","ALLOWED","ALLOWED",null,"ALLOWED","NOT ALLOWED",null,"NOT ALLOWED",null,null,"NOT ALLOWED","ALLOWED"],["ALLOWED","NOT ALLOWED","ALLOWED",null,null,"NOT ALLOWED",null,"NOT ALLOWED","ALLOWED","ALLOWED","ALLOWED",null,"ALLOWED","NOT ALLOWED",null,"NOT ALLOWED",null,null,"NOT ALLOWED","ALLOWED"],[null,"NOT REMOVE","NOT REMOVE",null,null,"NOT REMOVE",null,"NOT REMOVE","NOT REMOVE","NOT REMOVE","NOT REMOVE",null,"NOT REMOVE","REMOVE",null,"REMOVE",null,null,"NOT REMOVE","NOT REMOVE"],["NOT ALLOWED","ALLOWED","ALLOWED",null,null,"ALLOWED",null,null,"NOT ALLOWED","ALLOWED","NOT ALLOWED",null,"ALLOWED","NOT ALLOWED",null,"NOT ALLOWED",null,null,"ALLOWED","ALLOWED"],["NOT ALLOWED","NOT ALLOWED","NOT ALLOWED",null,null,"NOT ALLOWED",null,"NOT ALLOWED","NOT ALLOWED","ALLOWED","ALLOWED",null,"ALLOWED","NOT ALLOWED",null,"NOT ALLOWED",null,null,"NOT ALLOWED","NOT ALLOWED"],[null,"NOT REMOVE","NOT REMOVE",null,null,"NOT REMOVE",null,"REMOVE","REMOVE","NOT REMOVE",null,null,"NOT REMOVE","REMOVE",null,"REMOVE",null,null,"NOT REMOVE","REMOVE"],["NOT ALLOWED","ALLOWED","ALLOWED",null,null,"ALLOWED",null,"NOT ALLOWED","ALLOWED","ALLOWED",null,null,"ALLOWED","NOT ALLOWED",null,"NOT ALLOWED",null,null,"NOT ALLOWED","NOT ALLOWED"],["NOT FIRED","FIRED","NOT FIRED",null,null,null,null,"FIRED","FIRED","NOT FIRED",null,null,"NOT FIRED","FIRED",null,"FIRED",null,null,"NOT FIRED","NOT FIRED"],["NOT REMOVE","NOT REMOVE","NOT REMOVE",null,null,"NOT REMOVE",null,"REMOVE","NOT REMOVE","NOT REMOVE",null,null,"NOT REMOVE","REMOVE",null,"REMOVE",null,null,"NOT REMOVE","REMOVE"],["ALLOWED","ALLOWED","ALLOWED",null,null,"ALLOWED",null,"NOT ALLOWED","NOT ALLOWED","ALLOWED","ALLOWED",null,"ALLOWED","NOT ALLOWED",null,"NOT ALLOWED",null,null,"NOT ALLOWED","ALLOWED"],["NOT ALLOWED","NOT ALLOWED","ALLOWED",null,null,"NOT ALLOWED",null,null,"NOT ALLOWED","ALLOWED",null,null,"ALLOWED","NOT ALLOWED",null,"NOT ALLOWED",null,null,"NOT ALLOWED","ALLOWED"],["NOT REMOVE","NOT REMOVE","NOT REMOVE",null,null,"REMOVE",null,null,"REMOVE","NOT REMOVE",null,null,"NOT REMOVE","REMOVE",null,"REMOVE",null,null,"NOT REMOVE","NOT REMOVE"],["ALLOWED","NOT ALLOWED","ALLOWED",null,null,"ALLOWED",null,"ALLOWED","ALLOWED","ALLOWED",null,null,"ALLOWED",null,null,"ALLOWED",null,null,"ALLOWED","ALLOWED"],["ALLOWED","NOT ALLOWED","ALLOWED",null,null,"ALLOWED",null,"ALLOWED","ALLOWED","ALLOWED",null,null,"ALLOWED","NOT ALLOWED",null,"ALLOWED",null,null,"ALLOWED","ALLOWED"],["NOT REMOVE","NOT REMOVE","NOT REMOVE",null,null,"NOT REMOVE",null,"NOT REMOVE","REMOVE","NOT REMOVE",null,null,"NOT REMOVE","REMOVE",null,"NOT REMOVE",null,null,"REMOVE","NOT REMOVE"],["Not allowed","Not allowed","Yes, allowed",null,null,"Not allowed",null,"Not allowed","Not allowed","Yes, allowed",null,null,"Yes, allowed","Not allowed",null,"Not allowed",null,null,"Not allowed","Not allowed"],[null,"Not allowed","Yes, allowed",null,null,"Not allowed",null,"Not allowed","Not allowed","Not allowed",null,null,"Yes, allowed","Not allowed",null,"Not allowed",null,null,"Not allowed","Yes, allowed"],["Remove","Remove","Not remove",null,null,"Remove",null,"Remove","Remove","Not remove",null,null,"Remove","Remove",null,"Remove",null,null,"Remove","Not remove"],[null,"FAVOR",null,"FAVOR","OPPOSE","OPPOSE","FAVOR","OPPOSE",null,"OPPOSE","FAVOR","OPPOSE","OPPOSE",null,"OPPOSE","OPPOSE","FAVOR","OPPOSE","OPPOSE","FAVOR"],["OPPOSE","FAVOR","OPPOSE",null,null,"FAVOR",null,"FAVOR","FAVOR","FAVOR","FAVOR",null,"FAVOR","FAVOR",null,"FAVOR",null,null,"FAVOR","FAVOR"],[null,"NOT HARSH ENOUGH",null,null,"NOT HARSH ENOUGH","NOT HARSH ENOUGH","NOT HARSH ENOUGH","TOO HARSH","TOO HARSH","ABOUT RIGHT",null,null,"NOT HARSH ENOUGH","NOT HARSH ENOUGH","NOT HARSH ENOUGH","NOT HARSH ENOUGH",null,"NOT HARSH ENOUGH","NOT HARSH ENOUGH","NOT HARSH ENOUGH"],[null,"LEGAL","LEGAL","LEGAL","NOT LEGAL",null,"LEGAL","NOT LEGAL",null,"LEGAL",null,"LEGAL","LEGAL",null,"NOT LEGAL","NOT LEGAL","NOT LEGAL","LEGAL","LEGAL",null],["CATHOLIC","CATHOLIC","PROTESTANT","PROTESTANT","PROTESTANT","PROTESTANT","CHRISTIAN","CATHOLIC","NONE","NONE","CATHOLIC","PROTESTANT","NONE","PROTESTANT","CATHOLIC","PROTESTANT","CATHOLIC","PROTESTANT","CATHOLIC","CATHOLIC"],["MODERATE","MODERATE","MODERATE","LIBERAL","FUNDAMENTALIST",null,"MODERATE","MODERATE","LIBERAL","LIBERAL","MODERATE","FUNDAMENTALIST","LIBERAL","FUNDAMENTALIST","MODERATE","FUNDAMENTALIST","MODERATE","MODERATE","MODERATE","MODERATE"],["Once/yr","Never","Never","Never","<Once/yr","Never","Sev times/yr","Sev times/yr","Never","Once/yr","Every wk","Never","Never",">Once/wk","Sev times/yr","Once/mo","Every wk","2-3 times /mo","Once/mo","Once/yr"],["NOT VERY STRONG","NOT VERY STRONG","STRONG","NOT VERY STRONG","SOMEWHAT STRONG","NOT VERY STRONG","SOMEWHAT STRONG","NOT VERY STRONG","NO RELIGION","NO RELIGION","STRONG","NOT VERY STRONG","NO RELIGION","STRONG","NOT VERY STRONG","NOT VERY STRONG","SOMEWHAT STRONG","STRONG","NOT VERY STRONG","NOT VERY STRONG"],["YES","YES","YES","YES",null,"YES","YES","NO","NO",null,"YES","YES","YES","YES","YES","YES","YES","YES","NO","YES"],["ONCE A WEEK","NEVER","SEVERAL TIMES A DAY","NEVER","SEVERAL TIMES A DAY","SEVERAL TIMES A DAY","ONCE A WEEK","LT ONCE A WEEK","NEVER","ONCE A DAY","ONCE A DAY","ONCE A DAY","NEVER","SEVERAL TIMES A DAY","SEVERAL TIMES A DAY","SEVERAL TIMES A DAY","ONCE A WEEK","ONCE A DAY","ONCE A WEEK","NEVER"],["INSPIRED WORD","WORD OF GOD","INSPIRED WORD","INSPIRED WORD","WORD OF GOD","INSPIRED WORD","WORD OF GOD","INSPIRED WORD","INSPIRED WORD","INSPIRED WORD",null,"INSPIRED WORD","BOOK OF FABLES","WORD OF GOD","BOOK OF FABLES","INSPIRED WORD",null,"WORD OF GOD",null,"INSPIRED WORD"],[null,null,null,"OPPOSE PREF","OPPOSE PREF","OPPOSE PREF","OPPOSE PREF",null,"STRONGLY SUPPORT PREF",null,null,null,null,"STRONGLY SUPPORT PREF","STRONGLY SUPPORT PREF",null,"OPPOSE PREF","STRONGLY OPPOSE PREF",null,"SUPPORT PREF"],["AGREE SOMEWHAT",null,null,"AGREE STRONGLY","AGREE STRONGLY","AGREE SOMEWHAT","AGREE SOMEWHAT",null,"AGREE STRONGLY",null,null,"AGREE SOMEWHAT",null,"AGREE STRONGLY","AGREE SOMEWHAT",null,"NEITHER AGREE NOR DISAGREE","AGREE STRONGLY",null,"AGREE SOMEWHAT"],[5,6,5,null,null,5,null,5,9,7,9,null,5,5,null,9,null,null,7,7],[7,7,4,null,null,5,null,5,9,5,9,null,9,5,null,7,null,null,5,5],["VERY HAPPY","VERY HAPPY","PRETTY HAPPY","VERY HAPPY","VERY HAPPY","VERY HAPPY","PRETTY HAPPY","PRETTY HAPPY","NOT TOO HAPPY","PRETTY HAPPY","VERY HAPPY","PRETTY HAPPY","NOT TOO HAPPY","VERY HAPPY","NOT TOO HAPPY","PRETTY HAPPY","PRETTY HAPPY","PRETTY HAPPY","PRETTY HAPPY","PRETTY HAPPY"],[null,"HARDLY ANY","ONLY SOME","A GREAT DEAL","HARDLY ANY",null,"ONLY SOME","ONLY SOME",null,"ONLY SOME",null,"HARDLY ANY","HARDLY ANY",null,"ONLY SOME","ONLY SOME","HARDLY ANY","ONLY SOME","HARDLY ANY",null],[null,"A GREAT DEAL","ONLY SOME","A GREAT DEAL","HARDLY ANY",null,"HARDLY ANY","ONLY SOME",null,"HARDLY ANY",null,"HARDLY ANY","A GREAT DEAL",null,"A GREAT DEAL","ONLY SOME","ONLY SOME","ONLY SOME","HARDLY ANY",null],[null,"A GREAT DEAL","ONLY SOME","A GREAT DEAL","ONLY SOME",null,"ONLY SOME","ONLY SOME",null,"ONLY SOME",null,"HARDLY ANY","HARDLY ANY",null,"A GREAT DEAL","A GREAT DEAL","ONLY SOME","ONLY SOME","ONLY SOME",null],[null,"ONLY SOME","ONLY SOME","ONLY SOME","A GREAT DEAL",null,"HARDLY ANY","A GREAT DEAL",null,"ONLY SOME",null,"HARDLY ANY","ONLY SOME",null,"A GREAT DEAL","ONLY SOME","ONLY SOME","ONLY SOME","ONLY SOME",null],[null,"ONLY SOME","ONLY SOME","HARDLY ANY","A GREAT DEAL",null,"HARDLY ANY","HARDLY ANY",null,"A GREAT DEAL",null,"ONLY SOME","HARDLY ANY",null,"ONLY SOME","HARDLY ANY",null,"ONLY SOME","ONLY SOME",null],[null,"ONLY SOME","ONLY SOME","HARDLY ANY","HARDLY ANY",null,"ONLY SOME","ONLY SOME",null,"ONLY SOME",null,"ONLY SOME","ONLY SOME",null,"ONLY SOME","HARDLY ANY","ONLY SOME","ONLY SOME","HARDLY ANY",null],[null,"HARDLY ANY","ONLY SOME","ONLY SOME","HARDLY ANY",null,"ONLY SOME","ONLY SOME",null,"ONLY SOME",null,"HARDLY ANY","A GREAT DEAL",null,"ONLY SOME","ONLY SOME",null,"ONLY SOME","ONLY SOME",null],[null,"A GREAT DEAL","A GREAT DEAL","A GREAT DEAL","ONLY SOME",null,"ONLY SOME","A GREAT DEAL",null,"ONLY SOME",null,"ONLY SOME","A GREAT DEAL",null,"A GREAT DEAL","A GREAT DEAL","A GREAT DEAL","ONLY SOME","ONLY SOME",null],[null,"A GREAT DEAL","ONLY SOME","ONLY SOME","HARDLY ANY",null,"ONLY SOME","ONLY SOME",null,"ONLY SOME",null,"ONLY SOME","ONLY SOME",null,"ONLY SOME","ONLY SOME",null,"A GREAT DEAL","ONLY SOME",null],[null,"A GREAT DEAL","ONLY SOME","ONLY SOME","A GREAT DEAL",null,"ONLY SOME","A GREAT DEAL",null,"ONLY SOME",null,null,"HARDLY ANY",null,"A GREAT DEAL","ONLY SOME","ONLY SOME","ONLY SOME","ONLY SOME",null],[null,"A GREAT DEAL","A GREAT DEAL",null,"A GREAT DEAL",null,"ONLY SOME","ONLY SOME",null,"A GREAT DEAL",null,null,"ONLY SOME",null,"A GREAT DEAL","ONLY SOME",null,"ONLY SOME","ONLY SOME",null],[null,"A GREAT DEAL","HARDLY ANY","HARDLY ANY","HARDLY ANY",null,"ONLY SOME","ONLY SOME",null,"ONLY SOME",null,"HARDLY ANY","HARDLY ANY",null,"ONLY SOME","HARDLY ANY",null,"ONLY SOME","ONLY SOME",null],[null,"A GREAT DEAL","A GREAT DEAL","A GREAT DEAL","A GREAT DEAL",null,"A GREAT DEAL","ONLY SOME",null,"A GREAT DEAL",null,"A GREAT DEAL","ONLY SOME",null,"A GREAT DEAL","ONLY SOME","ONLY SOME","ONLY SOME","ONLY SOME",null],[null,"2ND IMPORTANT","4TH IMPORTANT","2ND IMPORTANT","4TH IMPORTANT",null,"3RD IMPORTANT","MOST IMPORTANT",null,"LEAST IMPORTANT",null,"4TH IMPORTANT","2ND IMPORTANT",null,"3RD IMPORTANT","3RD IMPORTANT","LEAST IMPORTANT","3RD IMPORTANT","3RD IMPORTANT",null],[null,"LEAST IMPORTANT","MOST IMPORTANT","LEAST IMPORTANT","LEAST IMPORTANT",null,"LEAST IMPORTANT","LEAST IMPORTANT",null,"4TH IMPORTANT",null,"LEAST IMPORTANT","LEAST IMPORTANT",null,"4TH IMPORTANT","LEAST IMPORTANT","4TH IMPORTANT","LEAST IMPORTANT","LEAST IMPORTANT",null],[null,"3RD IMPORTANT","LEAST IMPORTANT","3RD IMPORTANT","MOST IMPORTANT",null,"MOST IMPORTANT","4TH IMPORTANT",null,"2ND IMPORTANT",null,"MOST IMPORTANT","3RD IMPORTANT",null,"2ND IMPORTANT","MOST IMPORTANT","MOST IMPORTANT","2ND IMPORTANT","2ND IMPORTANT",null],[null,"MOST IMPORTANT","2ND IMPORTANT","MOST IMPORTANT","2ND IMPORTANT",null,"2ND IMPORTANT","2ND IMPORTANT",null,"3RD IMPORTANT",null,"3RD IMPORTANT","4TH IMPORTANT",null,"LEAST IMPORTANT","2ND IMPORTANT","2ND IMPORTANT","MOST IMPORTANT","MOST IMPORTANT",null],[null,"4TH IMPORTANT","3RD IMPORTANT","4TH IMPORTANT","3RD IMPORTANT",null,"4TH IMPORTANT","3RD IMPORTANT",null,"MOST IMPORTANT",null,"2ND IMPORTANT","MOST IMPORTANT",null,"MOST IMPORTANT","4TH IMPORTANT","3RD IMPORTANT","4TH IMPORTANT","4TH IMPORTANT",null],[null,"NEITHER BELONGS","NEITHER BELONGS","NEITHER BELONGS","NEITHER BELONGS",null,"NEITHER BELONGS","NEITHER BELONGS",null,"NEITHER BELONGS",null,"NEITHER BELONGS","NEITHER BELONGS",null,"NEITHER BELONGS","R BELONGS","R BELONGS","NEITHER BELONGS","R BELONGS",null],["BOTH EQUALLY","HARD WORK","LUCK OR HELP",null,null,"BOTH EQUALLY",null,"HARD WORK","BOTH EQUALLY","BOTH EQUALLY","HARD WORK",null,"HARD WORK","HARD WORK",null,"HARD WORK",null,null,"BOTH EQUALLY","HARD WORK"],["YES","YES","YES",null,null,"YES",null,"YES","YES","YES","YES",null,"YES","NO",null,"YES",null,null,"YES","YES"],[null,"YES","NO",null,null,"YES",null,"YES","NO","YES","NO",null,"NO","NO",null,"NO",null,null,"NO","NO"],["YES","YES","YES",null,null,"YES",null,"YES","YES","YES","YES",null,"YES","NO",null,"YES",null,null,"YES","YES"],[null,"YES","NO",null,null,"YES",null,"NO","NO","YES","NO",null,"NO","NO",null,"NO",null,null,"NO","NO"],["YES","YES","YES",null,null,"YES",null,"YES",null,"YES","NO",null,"YES","NO",null,"YES",null,null,"YES","YES"],[null,"YES","NO",null,null,"YES",null,"NO","NO","YES","NO",null,"NO","NO",null,"NO",null,null,"NO","NO"],["YES","YES","YES",null,null,"YES",null,"NO","NO","YES","NO",null,null,"NO",null,"YES",null,null,"NO","NO"],["MORE DIFFICULT",null,null,"STAY SAME","EASIER","EASIER","MORE DIFFICULT",null,"EASIER",null,null,"EASIER",null,"MORE DIFFICULT","EASIER",null,"MORE DIFFICULT","EASIER",null,"EASIER"],["NOT WRONG AT ALL",null,null,"NOT WRONG AT ALL","NOT WRONG AT ALL","NOT WRONG AT ALL","NOT WRONG AT ALL",null,"NOT WRONG AT ALL",null,null,"NOT WRONG AT ALL",null,"ALWAYS WRONG","NOT WRONG AT ALL",null,"ALWAYS WRONG","NOT WRONG AT ALL",null,"NOT WRONG AT ALL"],["ALMST ALWAYS WRG",null,null,"NOT WRONG AT ALL","ALWAYS WRONG","ALWAYS WRONG","ALWAYS WRONG",null,"SOMETIMES WRONG",null,null,null,null,"ALWAYS WRONG","ALWAYS WRONG",null,"ALWAYS WRONG","ALWAYS WRONG",null,"ALWAYS WRONG"],["ALWAYS WRONG","ALWAYS WRONG","ALMST ALWAYS WRG",null,null,"SOMETIMES WRONG",null,"ALWAYS WRONG","ALWAYS WRONG","ALMST ALWAYS WRG","SOMETIMES WRONG",null,"ALWAYS WRONG","ALWAYS WRONG",null,"ALWAYS WRONG",null,null,"ALWAYS WRONG","ALWAYS WRONG"],["NOT WRONG AT ALL","ALWAYS WRONG","NOT WRONG AT ALL",null,null,"NOT WRONG AT ALL",null,"NOT WRONG AT ALL","NOT WRONG AT ALL","NOT WRONG AT ALL",null,null,"NOT WRONG AT ALL","ALWAYS WRONG",null,"NOT WRONG AT ALL",null,null,"SOMETIMES WRONG","NOT WRONG AT ALL"],[null,"ILLEGAL UNDER 18","ILLEGAL UNDER 18","ILLEGAL UNDER 18","ILLEGAL UNDER 18",null,"ILLEGAL UNDER 18","LEGAL",null,"ILLEGAL UNDER 18",null,"ILLEGAL UNDER 18",null,null,"ILLEGAL UNDER 18","ILLEGAL UNDER 18","ILLEGAL UNDER 18","ILLEGAL UNDER 18","ILLEGAL TO ALL",null],["AGREE",null,null,"AGREE","STRONGLY AGREE",null,"AGREE",null,"STRONGLY AGREE",null,"AGREE","AGREE",null,"STRONGLY AGREE","DISAGREE",null,"AGREE","DISAGREE",null,"DISAGREE"],["NO","NO","NO",null,null,"NO",null,"NO","NO","NO","NO",null,"NO","NO",null,"NO",null,null,"NO","NO"],["ONCE A WEEK",null,null,"FEW TIMES A WEEK","FEW TIMES A WEEK","EVERYDAY","NEVER",null,"ONCE A WEEK",null,"EVERYDAY","NEVER",null,"LESS THAN ONCE WK","ONCE A WEEK",null,"EVERYDAY","ONCE A WEEK",null,"LESS THAN ONCE WK"],[1,null,null,4,3,4,3,null,null,null,6,2,null,2,2,null,2,3,null,5],["STRONGLY AGREE",null,null,"AGREE","AGREE","AGREE","AGREE",null,"AGREE",null,"AGREE","AGREE",null,"AGREE","DISAGREE",null,"DISAGREE","STRONGLY AGREE",null,"DISAGREE"],["AGREE",null,null,"DISAGREE","DISAGREE",null,"DISAGREE",null,"DISAGREE",null,"AGREE","STRONGLY DISAGREE",null,"STRONGLY AGREE","AGREE",null,"DISAGREE","DISAGREE",null,"DISAGREE"],["DISAGREE",null,null,"AGREE","DISAGREE","STRONGLY DISAGREE","DISAGREE",null,"DISAGREE",null,"AGREE","DISAGREE",null,"AGREE","AGREE",null,"DISAGREE","STRONGLY DISAGREE",null,"DISAGREE"],["YES",null,null,null,"NO","NO","NO",null,"YES",null,"YES","NO",null,"YES","YES",null,"NO","NO",null,"YES"],["NO",null,null,"NO","NO","NO","NO",null,"NO",null,null,"NO",null,"NO","YES",null,"NO","NO",null,"NO"],["NO",null,null,"YES","NO","NO","NO",null,"NO",null,null,"NO",null,"NO","YES",null,"NO","NO",null,"YES"],["NO",null,null,"YES","NO","YES","YES",null,"YES",null,null,"YES",null,"YES","YES",null,null,"YES",null,"NO"],[null,"Agree both","Agree both","Help selves","Govt",null,"Govt","Govt",null,"2",null,"Govt","Govt",null,"Help selves","Govt","Agree both","Agree both","Agree both",null],[null,2,3,5,3,null,3,1,null,1,null,3,3,null,1,1,3,3,3,null],[null,"4","4","Help selves","2",null,"Govt","Govt",null,"Govt",null,"Govt","Govt",null,"Govt","Govt","Agree both","Agree both","Govt",null],[null,5,4,5,4,null,5,3,null,2,null,3,2,null,2,1,5,3,3,null],[null,"Not at all interested","Moderately interested","Moderately interested","Moderately interested",null,null,"Not at all interested",null,"Very interested",null,null,"Not at all interested",null,"Moderately interested","Moderately interested","Not at all interested",null,"Moderately interested",null],[null,"Moderately interested","Not at all interested","Not at all interested","Not at all interested",null,null,"Moderately interested",null,"Moderately interested",null,null,"Moderately interested",null,"Not at all interested","Very interested","Moderately interested",null,"Moderately interested",null],[null,"Moderately interested","Moderately interested","Not at all interested","Very interested",null,null,"Very interested",null,"Moderately interested",null,null,"Very interested",null,"Not at all interested","Very interested","Very interested",null,"Very interested",null],[null,"Moderately interested","Moderately interested","Not at all interested","Very interested",null,null,"Moderately interested",null,"Moderately interested",null,null,"Moderately interested",null,"Very interested","Moderately interested",null,null,"Moderately interested",null],[null,"Moderately interested","Very interested","Very interested","Very interested",null,null,"Moderately interested",null,"Very interested",null,null,"Not at all interested",null,"Not at all interested","Very interested","Very interested",null,"Very interested",null],[null,"Very interested","Moderately interested","Moderately interested","Moderately interested",null,null,"Moderately interested",null,"Moderately interested",null,null,"Not at all interested",null,"Moderately interested","Very interested","Very interested",null,"Very interested",null],[null,"Moderately interested","Moderately interested","Very interested","Very interested",null,null,"Moderately interested",null,"Moderately interested",null,null,"Very interested",null,"Very interested","Very interested","Very interested",null,"Very interested",null],[null,"Moderately interested","Not at all interested","Not at all interested","Moderately interested",null,null,"Moderately interested",null,"Moderately interested",null,null,"Not at all interested",null,"Not at all interested","Moderately interested","Moderately interested",null,"Very interested",null],[null,"Very interested","Moderately interested","Moderately interested","Very interested",null,null,"Moderately interested",null,"Very interested",null,null,"Very interested",null,"Very interested","Moderately interested","Very interested",null,"Moderately interested",null],[null,"Very interested","Moderately interested","Not at all interested","Very interested",null,null,"Not at all interested",null,"Moderately interested",null,null,"Not at all interested",null,"Moderately interested","Moderately interested","Very interested",null,"Moderately interested",null],["NOT HISPANIC","NOT HISPANIC","DOMINICAN","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","MEXICAN, MEXICAN AMERICAN, CHICANO/A","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","DOMINICAN","NOT HISPANIC","MEXICAN, MEXICAN AMERICAN, CHICANO/A","NOT HISPANIC","DOMINICAN","DOMINICAN"],[null,null,null,null,null,null,null,"Not a U.S. Citizen",null,null,null,null,null,null,"Not a U.S. Citizen",null,"Not a U.S. Citizen",null,"Not a U.S. Citizen",null],["GERMANY","ITALY","WEST INDIES",null,"AFRICA",null,"GERMANY","MEXICO",null,"NORWAY",null,"AFRICA","POLAND","OTHER","OTHER SPANISH","AFRICA","MEXICO","AMERICAN ONLY","OTHER SPANISH","OTHER SPANISH"],[null,null,null,null,null,null,null,"Planning to apply for U.S. citizenship",null,null,null,null,null,null,"Planning to apply for U.S. citizenship",null,"Planning to apply for U.S. citizenship",null,"Not planning to apply to U.S. citizenship",null],["NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","LESS THAN 2 YRS","NONE"],[7,null,null,9,6,8,6,null,3,null,2,6,null,4,2,null,2,5,null,5],["MARRIED COUPLE, NO CHILDREN","MARRIED COUPLE W CHILDREN","MARRIED COUPLE W CHILDREN","MARRIED COUPLE, NO CHILDREN","SINGLE ADULT","COHAB COUPLE, NO CHILDREN","MARRIED COUPLE W CHILDREN","SINGLE PARENT","MARRIED COUPLE, NO CHILDREN","SINGLE ADULT","UNSURE, NO CHILDREN","COHAB COUPLE W CHILDREN","SINGLE PARENT","SINGLE ADULT","UNSURE W CHILDREN","SINGLE PARENT","MARRIED COUPLE W CHILDREN","OTHER FAMILY W CHILDREN","MARRIED COUPLE W CHILDREN","UNSURE W CHILDREN"],[721,624,627,527,620,502,421,405,404,517,404,409,404,502,517,622,411,612,325,320],[1990,1991,1970,1963,1942,1962,1977,1988,1984,1984,1957,1976,1984,1953,1960,1977,1976,1965,1957,1994],["LIBRA","ARIES","ARIES","CAPRICORN","TAURUS","TAURUS","SCORPIO","SCORPIO","LIBRA","TAURUS","PISCES","CAPRICORN","CAPRICORN","SAGITTARIUS","SAGITTARIUS","PISCES","GEMINI","GEMINI","TAURUS","PISCES"],["NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC",null,"NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC","NOT HISPANIC"],["White","White","White","White","White","White","White","White","White","White","White","White","White","White",null,"White","White","White","Black or African","Black or African"],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],["YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","YES, MONEY","NO","YES, MONEY","YES, MONEY","YES, MONEY","NO","NO"],[75,75,75,20,75,20,20,20,20,20,20,20,20,20,null,75,20,50,null,null],[69,53,77,78,149,98,80,74,88,76,96,91,95,121,165,64,105,81,115,94],[60,32,32,60,62,62,62,62,62,62,60,60,60,60,56,32,60,32,40,40],["WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","HISPANIC","WHITE","WHITE","WHITE","BLACK","BLACK"],["IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","OVER THE PHONE","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON","IN-PERSON"],["FEMALE","FEMALE","FEMALE","FEMALE","FEMALE","FEMALE","FEMALE","FEMALE","FEMALE","FEMALE","FEMALE","FEMALE","FEMALE","FEMALE","MALE","FEMALE","FEMALE","FEMALE","FEMALE","FEMALE"],[2,0,0,2,2,2,2,2,2,2,2,2,2,2,0,0,2,0,2,2],[2.6219628630005,3.4959504840006,1.7479752420003,1.235694389852,0.8739876210002,0.8237962599014,0.8237962599014,0.4118981299507,0.4118981299507,0.4118981299507,0.8237962599014,0.8237962599014,0.4118981299507,0.4118981299507,1.235694389852,0.8739876210002,0.8237962599014,1.7479752420003,0.8237962599014,1.6475925198027],[null,0,2,3,0,null,0,0,null,0,null,1,null,null,0,0,1,1,2,null],[null,4,2,1,4,null,2,3,null,4,null,null,1,null,4,1,null,3,3,null],[8,null,null,1,5,6,10,null,5,null,5,8,null,7,1,null,10,4,null,5],[null,7,4,null,null,7,null,4,null,7,2,null,null,0,null,4,null,null,3,3],[null,"6-7","3-5",null,null,"6-7",null,"3-5",null,"6-7","0-2",null,null,"0-2",null,"3-5",null,null,"3-5","3-5"],[null,4,4,4,1,null,2,6,null,1,null,1,4,null,3,2,0,3,3,null],[6,null,null,5,6,null,6,null,6,null,4,7,null,3,3,null,5,8,null,5],["No","No","No","No","No","No","Yes","No","No","Yes",null,"No","No","Yes","Yes","Yes","No","Yes",null,"No"],["No","No","Yes","No","Yes","Yes","Yes","No","No","Yes",null,"No","No","Yes","Yes","No","No","No","No","No"],["No","No","No","No","Yes","No","Yes","Yes","Yes","No","Yes","Yes","No","Yes","Yes","Yes","Yes","Yes",null,"No"],[null,7,10,null,4,null,null,5,null,7,null,null,null,null,6,null,null,null,null,null],["Coll+","HS","HS","Some Coll","Coll+","Coll+","Some Coll","<HS","<HS","Coll+","<HS","Coll+","HS","HS","<HS","Some Coll","HS","Some Coll","HS","HS"],["13+ yrs","0-12 yrs","0-12 yrs","13+ yrs","13+ yrs","13+ yrs","13+ yrs","0-12 yrs","0-12 yrs","13+ yrs","0-12 yrs","13+ yrs","0-12 yrs","0-12 yrs","0-12 yrs","13+ yrs","0-12 yrs","13+ yrs","0-12 yrs","0-12 yrs"],["No","No","Yes","Yes","Yes","Yes","Yes","Yes","Yes","No","Yes","Yes","Yes","Yes","Yes","Yes","Yes","Yes","Yes","No"],["18-30","18-30","41-50","41-50","60-","41-50","31-40","18-30","18-30","18-30","51-60","31-40","18-30","51-60","51-60","31-40","31-40","41-50","51-60","18-30"],["Low","Low","Mod","Low","Mod","Mod","High","Low","Low","High",null,"Low","Low","High","High","Mod","Low","Mod",null,"Low"],["White","White","White","White","White","White","White","White","White","White","White","White","White","White",null,"White","White","White","Black","Black"],["Less close","More close","Less close",null,null,"Less close",null,"Less close","More close","More close","More close",null,"Less close","Less close",null,"More close",null,null,"More close","More close"],["More close","More close","Less close",null,null,"Less close",null,"Less close","More close","Less close","More close",null,"More close","Less close",null,"More close",null,null,"Less close","Less close"],[null,"More important","Less important","More important","Less important",null,"Less important","More important",null,"Less important",null,"Less important","More important",null,"Less important","Less important","Less important","Less important","Less important",null],["No","No","No","No","Yes",null,"No","No","No","No","No","Yes","No","Yes","No","Yes","No","No","No","No"],["Low","Low","Low","Low","Low","Low","Moderate","Moderate","Low","Low","High","Low","Low","High","Moderate","Moderate","High","Moderate","Moderate","Low"],["No","Yes","No",null,null,"No",null,"No","No","No",null,null,"No","Yes",null,"No",null,null,"No","No"],["Not strong","Not strong","Strong","Not strong","Not strong","Not strong","Not strong","Not strong","Not strong","Not strong","Strong","Not strong","Not strong","Strong","Not strong","Not strong","Not strong","Strong","Not strong","Not strong"],[null,null,null,"Oppose","Oppose","Oppose","Oppose",null,"Support",null,null,null,null,"Support","Support",null,"Oppose","Oppose",null,"Support"],["Mod","Conserv","Conserv","Conserv","Conserv","Conserv","Conserv","Conserv","Liberal","Mod","Mod","Mod","Conserv","Liberal","Liberal","Mod","Conserv","Conserv","Conserv","Liberal"],["Liberal","Mod","Liberal","Liberal","Liberal","Liberal","Liberal","Mod","Liberal","Liberal","Mod","Liberal","Liberal","Mod","Liberal","Liberal","Mod","Liberal","Mod","Mod"],["Conserv","Conserv","Mod","Conserv","Conserv","Liberal","Liberal","Conserv","Liberal","Liberal","Mod","Liberal","Conserv","Liberal","Conserv","Liberal","Conserv","Liberal","Mod","Conserv"],["No","No","Yes",null,null,"No",null,"No","No","Yes",null,null,"No","No",null,"No",null,null,"No","Yes"],["No","No","Yes",null,null,"No",null,"No","No","Yes",null,null,"Yes","No",null,"No",null,null,"No","No"],[null,"No","Yes",null,null,"No",null,"No","No","No",null,null,"Yes","No",null,"No",null,null,"No","Yes"],[null,null,"High",null,null,null,null,null,null,"Mid",null,"Low",null,null,"MedLow","MedHigh","MedHigh","MedHigh","MedHigh",null],[null,22,14,1,20,null,34,35,null,33,null,35,30,null,1,35,18,29,7,null],["White","White",null,"White","Black","White","White",null,"Black","White","White","Black","White","Black",null,"Black","White","Black",null,null],[0,null,null,0,1,4,null,null,null,null,3,0,null,0,6,null,1,1,null,1],[0,null,null,0,0,-1,null,null,null,null,null,0,null,0,3,null,1,-1,null,0],[null,12,9,7,15,null,null,9,null,13,null,null,8,null,9,15,null,null,15,null],["NonTrad",null,null,"Traditional","NonTrad",null,"NonTrad",null,"NonTrad",null,"Traditional","NonTrad",null,"Traditional","Traditional",null,"Traditional","NonTrad",null,"Traditional"],[null,10,13,null,null,null,null,null,8,14,null,null,14,null,null,4,null,null,7,10],["Middle","Middle","High",null,null,"Low",null,null,"Low","High",null,null,"High","Low",null,"Low",null,null,"Low","High"],[null,"Middle","Less egal","Less egal","Middle",null,"More egal","More egal",null,"More egal",null,"More egal","More egal",null,"Less egal","More egal","Middle","More egal","Less egal",null],["No","No",null,"No","Yes","No","No",null,"Yes","No","No","Yes","No","Yes",null,"Yes","No","Yes",null,null],["Did not vote",null,"Voted","Voted","Voted","Voted","Did not vote",null,"Voted","Voted","Voted","Did not vote","Did not vote","Voted","Did not vote","Voted",null,"Voted","Did not vote",null],["Ind","Ind","Ind","Rep","Dem","Dem","Ind","Ind","Ind","Dem","Dem","Dem","Ind","Ind","Ind","Rep","Rep","Ind",null,"Ind"],["No","No","Yes","No","No","No","No","Yes","No","No","No","No","No","No","Yes","No","Yes","No","Yes","Yes"],[2.6219628630005,3.4959504840006,1.7479752420003,1.235694389852,0.8739876210002,0.8237962599014,0.8237962599014,0.4118981299507,0.4118981299507,0.4118981299507,0.8237962599014,0.8237962599014,0.4118981299507,0.4118981299507,1.235694389852,0.8739876210002,0.8237962599014,1.7479752420003,0.8237962599014,1.6475925198027],["Once/yr","Never/<1yr","Never/<1yr","Never/<1yr","Never/<1yr","Never/<1yr","Svrl times/yr","Svrl times/yr","Never/<1yr","Once/yr","Ev wk or more","Never/<1yr","Never/<1yr","Ev wk or more","Svrl times/yr","Once/mnth","Ev wk or more","2-3 times/mnth","Once/mnth","Once/yr"],[null,"MedHigh","MedLow","Low","MedLow",null,"High","High",null,"High",null,"High","High",null,"Low","High","MedLow","MedHigh","Low",null],[null,"MedLow","MedHigh",null,null,null,null,null,"Low","MedHigh",null,null,"MedHigh",null,null,"Low",null,null,"Low","MedLow"],["Approve",null,null,"Approve","Approve",null,"Approve",null,"Approve",null,"Approve","Approve",null,"Approve","Disapprove",null,"Approve","Disapprove",null,"Disapprove"],["18-40","18-40","41-60","41-60","61+","41-60","18-40","18-40","18-40","18-40","41-60","18-40","18-40","41-60","41-60","18-40","18-40","41-60","41-60","18-40"],["0-1",null,null,"4","3","4","3",null,null,null,"5 or more","2",null,"2","2",null,"2","3",null,"5 or more"],["Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","High","Low","Low","High","Low","High","High","High","High","Low"],[null,"Not illegal to all","Not illegal to all","Not illegal to all","Not illegal to all",null,"Not illegal to all","Not illegal to all",null,"Not illegal to all",null,"Not illegal to all",null,null,"Not illegal to all","Not illegal to all","Not illegal to all","Not illegal to all","Illegal to all",null],["Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth","Nonsouth"],["More difficult",null,null,"Same/easier","Same/easier","Same/easier","More difficult",null,"Same/easier",null,null,"Same/easier",null,"More difficult","Same/easier",null,"More difficult","Same/easier",null,"Same/easier"],["22","21","42","49","70","50","35","24","28","28","55","36","28","59","52","35","36","47","55","18"],[0,0,1,null,null,0,null,0,0,1,null,null,1,0,null,0,null,null,0,0],[0,0,100,null,null,0,null,0,0,100,null,null,100,0,null,0,null,null,0,0],[null,100,100,100,0,null,100,0,null,100,null,100,100,null,0,0,0,100,100,null],["<=30","<=30",">=31",">=31",">=31",">=31",">=31","<=30","<=30","<=30",">=31",">=31","<=30",">=31",">=31",">=31",">=31",">=31",">=31","<=30"],["Catholic","Catholic","Protestant","Protestant","Protestant","Protestant",null,"Catholic",null,null,"Catholic","Protestant",null,"Protestant","Catholic","Protestant","Catholic","Protestant","Catholic","Catholic"],[0,0,0,0,1,1,0,0,0,1,1,1,0,0,0,0,0,0,null,0],[null,"7","10",null,"4",null,null,"5",null,"7",null,null,null,null,"6",null,null,null,null,null]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>year<\/th>\n      <th>id<\/th>\n      <th>wrkstat<\/th>\n      <th>wrkslf<\/th>\n      <th>wrkgvt<\/th>\n      <th>marital<\/th>\n      <th>sibs<\/th>\n      <th>childs<\/th>\n      <th>age<\/th>\n      <th>educ<\/th>\n      <th>degree<\/th>\n      <th>sex<\/th>\n      <th>race<\/th>\n      <th>polviews<\/th>\n      <th>partyid<\/th>\n      <th>mobile16<\/th>\n      <th>born<\/th>\n      <th>income06<\/th>\n      <th>rincom06<\/th>\n      <th>region<\/th>\n      <th>size<\/th>\n      <th>vote08_coded<\/th>\n      <th>pres08<\/th>\n      <th>natspac<\/th>\n      <th>natenvir<\/th>\n      <th>natheal<\/th>\n      <th>natcity<\/th>\n      <th>natcrime<\/th>\n      <th>natdrug<\/th>\n      <th>nateduc<\/th>\n      <th>natrace<\/th>\n      <th>natarms<\/th>\n      <th>nataid<\/th>\n      <th>natfare<\/th>\n      <th>natroad<\/th>\n      <th>natsoc<\/th>\n      <th>natmass<\/th>\n      <th>natpark<\/th>\n      <th>natchld<\/th>\n      <th>natsci<\/th>\n      <th>natenrgy<\/th>\n      <th>eqwlth<\/th>\n      <th>spkath<\/th>\n      <th>colath<\/th>\n      <th>libath<\/th>\n      <th>spkrac<\/th>\n      <th>colrac<\/th>\n      <th>librac<\/th>\n      <th>spkcom<\/th>\n      <th>colcom<\/th>\n      <th>libcom<\/th>\n      <th>spkmil<\/th>\n      <th>colmil<\/th>\n      <th>libmil<\/th>\n      <th>spkhomo<\/th>\n      <th>colhomo<\/th>\n      <th>libhomo<\/th>\n      <th>spkmslm<\/th>\n      <th>colmslm<\/th>\n      <th>libmslm<\/th>\n      <th>cappun<\/th>\n      <th>gunlaw<\/th>\n      <th>courts<\/th>\n      <th>grass<\/th>\n      <th>relig<\/th>\n      <th>fund<\/th>\n      <th>attend<\/th>\n      <th>reliten<\/th>\n      <th>postlife<\/th>\n      <th>pray<\/th>\n      <th>bible<\/th>\n      <th>affrmact<\/th>\n      <th>wrkwayup<\/th>\n      <th>closeblk<\/th>\n      <th>closewht<\/th>\n      <th>happy<\/th>\n      <th>confinan<\/th>\n      <th>conbus<\/th>\n      <th>conclerg<\/th>\n      <th>coneduc<\/th>\n      <th>confed<\/th>\n      <th>conlabor<\/th>\n      <th>conpress<\/th>\n      <th>conmedic<\/th>\n      <th>contv<\/th>\n      <th>conjudge<\/th>\n      <th>consci<\/th>\n      <th>conlegis<\/th>\n      <th>conarmy<\/th>\n      <th>obey<\/th>\n      <th>popular<\/th>\n      <th>thnkself<\/th>\n      <th>workhard<\/th>\n      <th>helpoth<\/th>\n      <th>union<\/th>\n      <th>getahead<\/th>\n      <th>abdefect<\/th>\n      <th>abnomore<\/th>\n      <th>abhlth<\/th>\n      <th>abpoor<\/th>\n      <th>abrape<\/th>\n      <th>absingle<\/th>\n      <th>abany<\/th>\n      <th>divlaw<\/th>\n      <th>premarsx<\/th>\n      <th>teensex<\/th>\n      <th>xmarsex<\/th>\n      <th>homosex<\/th>\n      <th>pornlaw<\/th>\n      <th>spanking<\/th>\n      <th>owngun<\/th>\n      <th>news<\/th>\n      <th>tvhours<\/th>\n      <th>fechld<\/th>\n      <th>fepresch<\/th>\n      <th>fefam<\/th>\n      <th>racdif1<\/th>\n      <th>racdif2<\/th>\n      <th>racdif3<\/th>\n      <th>racdif4<\/th>\n      <th>helppoor<\/th>\n      <th>helpnot<\/th>\n      <th>helpsick<\/th>\n      <th>helpblk<\/th>\n      <th>intintl<\/th>\n      <th>intfarm<\/th>\n      <th>inteduc<\/th>\n      <th>intsci<\/th>\n      <th>intecon<\/th>\n      <th>inttech<\/th>\n      <th>intmed<\/th>\n      <th>intspace<\/th>\n      <th>intenvir<\/th>\n      <th>intmil<\/th>\n      <th>hispanic<\/th>\n      <th>uscitzn<\/th>\n      <th>ethnic<\/th>\n      <th>fucitzn<\/th>\n      <th>vetyears<\/th>\n      <th>wordsum<\/th>\n      <th>hhtype1<\/th>\n      <th>dateintv<\/th>\n      <th>cohort<\/th>\n      <th>zodiac<\/th>\n      <th>inthisp<\/th>\n      <th>intrace1<\/th>\n      <th>intrace2<\/th>\n      <th>intrace3<\/th>\n      <th>feeused<\/th>\n      <th>feelevel<\/th>\n      <th>lngthinv<\/th>\n      <th>intage<\/th>\n      <th>intethn<\/th>\n      <th>mode<\/th>\n      <th>intsex<\/th>\n      <th>intyrs<\/th>\n      <th>wtss<\/th>\n      <th>social_trust<\/th>\n      <th>con_govt<\/th>\n      <th>social_connect<\/th>\n      <th>abortion<\/th>\n      <th>abortion_3<\/th>\n      <th>authoritarianism<\/th>\n      <th>femrole<\/th>\n      <th>reborn_r<\/th>\n      <th>savesoul_r<\/th>\n      <th>god_r<\/th>\n      <th>science_quiz<\/th>\n      <th>educ_4<\/th>\n      <th>educ_2<\/th>\n      <th>kids<\/th>\n      <th>age_5<\/th>\n      <th>evangelical<\/th>\n      <th>intethn_2<\/th>\n      <th>closeblk2<\/th>\n      <th>closewht2<\/th>\n      <th>obey2<\/th>\n      <th>fund2<\/th>\n      <th>attend3<\/th>\n      <th>homosex2<\/th>\n      <th>reliten2<\/th>\n      <th>affrmact2<\/th>\n      <th>racial_liberal3<\/th>\n      <th>social_cons3<\/th>\n      <th>spend3<\/th>\n      <th>mslm_lib<\/th>\n      <th>mslm_spk<\/th>\n      <th>mslm_col<\/th>\n      <th>rincom06_5<\/th>\n      <th>egalit_scale<\/th>\n      <th>race_2<\/th>\n      <th>white_traits<\/th>\n      <th>black_traits<\/th>\n      <th>int_info_scale<\/th>\n      <th>femrole2<\/th>\n      <th>tolerance<\/th>\n      <th>militarist_tol<\/th>\n      <th>egalit_scale3<\/th>\n      <th>black<\/th>\n      <th>voted08<\/th>\n      <th>partyid_3<\/th>\n      <th>hispanic_2<\/th>\n      <th>wtssall<\/th>\n      <th>attend7<\/th>\n      <th>egalit_scale4<\/th>\n      <th>tolerance4<\/th>\n      <th>spanking2<\/th>\n      <th>age_3<\/th>\n      <th>tvhours5<\/th>\n      <th>attend2<\/th>\n      <th>pornlaw2<\/th>\n      <th>south<\/th>\n      <th>divlaw2<\/th>\n      <th>age.f<\/th>\n      <th>mslm_spk.high<\/th>\n      <th>mslm_spk.yes<\/th>\n      <th>grass.legal<\/th>\n      <th>age2<\/th>\n      <th>relig2<\/th>\n      <th>dem<\/th>\n      <th>science_quiz.f<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"pageLength":6,"columnDefs":[{"className":"dt-right","targets":[1,2,7,8,9,21,42,74,75,113,122,124,140,142,143,150,151,152,156,157,158,159,160,161,163,164,168,190,192,193,194,196,203,215,216,217,220]},{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[6,10,25,50,100]}},"evals":[],"jsHooks":[]}</script>

---
### Prediction Workflow Example

__Step 1:__ We're going to randomly assign 80% of our data to be the "training set" and the remaining 20% of our data as the "test set".

This leaves our training data with 1580 cases and our testing data with 394 cases.

--
__Step 2:__ We can estimate a model using our training data (and only our training data)

`$$P(Vote = 1) = \Lambda(\beta_0 + \beta_1Age + \beta_2Educ + \beta_3Sex + \beta_4Race + \beta_5Party + \beta_6Income)$$`

--
__Step 3:__ We will use that model estimated in step 2 and make predictions to our test data

---
### Prediction Workflow Example

__Step 4:__ Assess the accuracy of the model!

Since we're using a simple logit model, let's use a different metric (more on this later).

`$$Accuracy = \frac{Correct Predictions}{All Predictions}$$`
`$$Accuracy = \frac{Correct Predictions}{All Predictions}$$`

`$$Accuracy = .745$$`

---
### K-fold Cross Validation
__K-fold cross-validation__ builds on the validation method and divides the dataset into `$k$` number of subsets and computes our performance metric K number of times using each subset as a testing set.

For example, imagine we have a model and dataset that we want to use k-fold cross-validation where `$k = 5$`.

1. Randomly split the data into 5 subsets (A, B, C, D, and E). 
2. For the first iteration, hold out subset A as the test set and use the remaining subsets (B, C, D, and E) as the training set. 
3. Test the model that you developed on the remaining subsets on the test set and store the diagnostic we care about (e.g. prediction accuracy).
4. Repeat steps 2 and 3 for each subset. 
5. Average the stored diagnostics and report it as the "cross validation performance metric."

--
_Why is this better than the validation set method?:_

More robust due to repeated randomization.

--
_How do we decide `$k$`?_

Typically, you will see k as 5 or 10 as those figures have been seen to not sacrifice bias at the expense of variance (or vice versa). i.e. Lower k values may be more biased but higher k values may have large variability.

---
### K-fold Cross Validation (K = 5)
<img src="Week12_files/figure-html/unnamed-chunk-22-1.png" height="80%" style="display: block; margin: auto;" />

---
### K-fold Cross Validation (K = 5)
<img src="Week12_files/figure-html/unnamed-chunk-23-1.png" height="80%" style="display: block; margin: auto;" />

---
### K-fold Cross Validation (K = 5)
<img src="Week12_files/figure-html/unnamed-chunk-24-1.png" height="80%" style="display: block; margin: auto;" />

---
### K-fold Cross Validation (K = 5)
<img src="Week12_files/figure-html/unnamed-chunk-25-1.png" height="80%" style="display: block; margin: auto;" />
---
### K-fold Cross Validation (K = 5)
<img src="Week12_files/figure-html/unnamed-chunk-26-1.png" height="80%" style="display: block; margin: auto;" />

---
### K-fold Cross Validation (K = 5)
<img src="Week12_files/figure-html/unnamed-chunk-27-1.png" height="80%" style="display: block; margin: auto;" />

---
### K-fold Cross Validation (K = 5)
<img src="Week12_files/figure-html/unnamed-chunk-28-1.png" height="80%" style="display: block; margin: auto;" />

---
### K-fold Cross Validation (K = 5)
<img src="Week12_files/figure-html/unnamed-chunk-29-1.png" height="80%" style="display: block; margin: auto;" />

---
### Model Metrics

With the models that we've introduced in class (OLS, logit, probit), there are a number of different model metrics that we can use to assess model performance (think: goodness of fit). We're only going to scratch the surface here (and you don't need to memorize these at all) but...

__OLS__ _(Continuous DV)_

`$R^2$` or `$Adj. R^2$`

`$RMSE$` (or just `$MSE$`)

`$MAE$` or Mean Absolute Error, `$= \frac{|y_i - y_p|}{n}$`

`$MAPE$` or Mean Absolute Percentage Error, `$= \frac{100\%}{n}\sum\frac{|y_i - y_p|}{y_i}$`

---
### Model Metrics

__Logit/Probit__ _(Binary DV)_

`$Pseudo R^2$`

`$Accuracy$` - Correct Predictions

`$Specificty$` - Correctly Identified Negative Cases

`$Sensitivity$` - Correctly Identified Positive Cases

`$Brier Score$` - Evaluating the probability of the event, `$= \frac{1}{N}\sum(p_i - y_i)^2$`

Each has different benefits and disadvantages which we we'll pick up again during the Stata session.

---
### Applications

So why do we want to do this again?

Predictive models can get quite complicated but can begin with all the models that we've been estimating all semester long.

There are a lot of applications, many of which are explicitly working their way into the research sphere. Some examples include:

* developing microtargeting predictions for voters, political supporters, etc.
* working to predict outcomes of various policy implementations (think Covid vaccine policies)
* identifying opinion changes from smaller studies and extrapolating to broader populations

Though as models get more complicated, we have to be clear to think through the unintended consequences of our predictive models.

---
## Where We're Going Next

Missing data, and other data concerns

__Reading__

* Wooldridge: 9-4, 9-5