IV II &
RDD II

Session 12

PMAP 8521: Program evaluation
Andrew Young School of Policy Studies

1 / 50

Plan for today2 / 50

Plan for today

Treatment effects and compliance

2 / 50

Plan for today

Treatment effects and compliance

Randomized promotion

2 / 50

Plan for today

Treatment effects and compliance

Randomized promotion

Fuzzy regression discontinuity

2 / 50

Treatment effects
and compliance3 / 50

Potential outcomes

$\delta = (Y |\ P = 1) - (Y |\ P = 0)$

4 / 50

Potential outcomes

$\delta = (Y |\ P = 1) - (Y |\ P = 0)$

δ (delta) = causal effect

4 / 50

Potential outcomes

$\delta = (Y |\ P = 1) - (Y |\ P = 0)$

δ (delta) = causal effect

P = Program

4 / 50

Potential outcomes

$\delta = (Y |\ P = 1) - (Y |\ P = 0)$

δ (delta) = causal effect

P = Program

Y = Outcome

4 / 50

Potential outcomes

$\delta = (Y |\ P = 1) - (Y |\ P = 0)$

δ (delta) = causal effect

P = Program

Y = Outcome

$\delta = Y_1 - Y_0$

4 / 50

Fundamental problem
of causal inference

5 / 50

Fundamental problem
of causal inference

$\delta_i = Y_i^1 - Y_i^0 \quad \text{in real life is} \quad \delta_i = Y_i^1 - ???$

5 / 50

Fundamental problem
of causal inference

$\delta_i = Y_i^1 - Y_i^0 \quad \text{in real life is} \quad \delta_i = Y_i^1 - ???$

Individual-level effects are
impossible to observe!

5 / 50

Average treatment effect

Difference between average/expected value when
program is on vs. expected value when program is off

$\text{ATE} = E(Y_1 - Y_0) = E(Y_1) - E(Y_0)$

6 / 50

Average treatment effect

Difference between average/expected value when
program is on vs. expected value when program is off

$\text{ATE} = E(Y_1 - Y_0) = E(Y_1) - E(Y_0)$

Can be found for a whole population, on average

$\delta = (\bar{Y}\ |\ P = 1) - (\bar{Y}\ |\ P = 0)$

6 / 50

Every individual has a
treatment/causal effect

7 / 50

Every individual has a
treatment/causal effect

ATE = average of all
unit-level causal effects

7 / 50

Every individual has a
treatment/causal effect

ATE = average of all
unit-level causal effects

ATE = Average effect
for the whole population

7 / 50

Other versions of causal effects

Average treatment on the treated

ATT/TOT

8 / 50

Other versions of causal effects

Average treatment on the treated

ATT/TOT

Conditional average treatment effect

CATE

8 / 50

Local effects

9 / 50

LATE

Local average treatment effect (LATE) =
weighted ATE

Narrower effect; only applies to some of the population

10 / 50

LATE

Local average treatment effect (LATE) =
weighted ATE

Narrower effect; only applies to some of the population

You can't make population-level
claims with LATE

10 / 50

LATE

Local average treatment effect (LATE) =
weighted ATE

Narrower effect; only applies to some of the population

You can't make population-level
claims with LATE

(But that can be okay!)

10 / 50

LATE

11 / 50

LATE

In RDD, LATE = people in the bandwidth

11 / 50

LATE

In RDD, LATE = people in the bandwidth

In RCTs and IVs, LATE = compliers

11 / 50

Compliance

Complier

Treatment
follows assignment

Always taker

Gets treatment
regardless of assignment

12 / 50

Compliance

Complier

Treatment
follows assignment

Always taker

Gets treatment
regardless of assignment

Never taker

Rejects treatment
regardless of assignment

Defier

Does the opposite
of assignment

12 / 50

13 / 50

Ignoring defiers14 / 50

Ignoring defiers

We can generally assume
that defiers don't exist

14 / 50

Ignoring defiers

We can generally assume
that defiers don't exist

In drug trials this makes sense; you can't get access
to medicine without being in treatment group

14 / 50

Ignoring defiers

We can generally assume
that defiers don't exist

In drug trials this makes sense; you can't get access
to medicine without being in treatment group

In development it can make sense; in a bed net RCT,
a defier assigned to treatment would have to
tear down all existing bed nets out of spite

14 / 50

Ignoring defiers

15 / 50

Ignoring defiers

Monotonicity assumption

15 / 50

Ignoring defiers

Monotonicity assumption

Assignment to treatment only
has an effect in one direction

15 / 50

Ignoring defiers

Monotonicity assumption

Assignment to treatment only
has an effect in one direction

Assignment to treatment can only
increase—not decrease—your actual chance of treatment

15 / 50

16 / 50

More causal effects

Intent to treat (ITT)

Effect of assignment (not actual treatment!)

17 / 50

More causal effects

Intent to treat (ITT)

Effect of assignment (not actual treatment!)

17 / 50

More causal effects

Complier Average Causal Effect (CACE)

LATE for the compliers

18 / 50

More causal effects

Complier Average Causal Effect (CACE)

LATE for the compliers

18 / 50

Hypothetical bed net program

An NGO distributes mosquito bed nets to help
improve health by reducing malaria infection rate

19 / 50

Hypothetical bed net program

An NGO distributes mosquito bed nets to help
improve health by reducing malaria infection rate

We can read everyone's minds and we know if
people are always takers, never takers, or compliers

19 / 50

Mind reading

20 / 50

Actual data

But we can't read minds! This is what we actually see:

21 / 50

Actual data

(Actually this is what we see)

22 / 50

23 / 50

$\begin{aligned} \text{ITT}\ =\ & \color{#0D0887}{\pi_\text{compliers} \times (\text{T} - \text{C})_\text{compliers}} + \\ &\color{#B7318A}{\pi_\text{always takers} \times (\text{T} - \text{C})_\text{always takers}} + \\ &\color{#FEBA2C}{\pi_\text{never takers} \times (\text{T} - \text{C})_\text{never takers}}\\[6pt] \text{ITT}\ =\ & \color{#0D0887}{\pi_\text{C} \text{CACE}} + \color{#B7318A}{\pi_\text{A} \text{ATACE}} + \color{#FEBA2C}{\pi_\text{N} \text{NTACE}} \end{aligned}$

24 / 50

$\text{ITT}\ =\ \color{#0D0887}{\pi_\text{C} \text{CACE}} + \color{#B7318A}{\pi_\text{A} \text{ATACE}} + \color{#FEBA2C}{\pi_\text{N} \text{NTACE}}$

25 / 50

$\text{ITT}\ =\ \color{#0D0887}{\pi_\text{C} \text{CACE}} + \color{#B7318A}{\pi_\text{A} \text{ATACE}} + \color{#FEBA2C}{\pi_\text{N} \text{NTACE}}$ Treatment received is same regardless of assignment!
Being assigned to treatment doesn't influence ATs and NTs

25 / 50

$\text{ITT}\ =\ \color{#0D0887}{\pi_\text{C} \text{CACE}} + \color{#B7318A}{\pi_\text{A} \times 0} + \color{#FEBA2C}{\pi_\text{N} \times 0}$

25 / 50

$\begin{aligned} \text{ITT}\ =\ & \color{#0D0887}{\pi_\text{C} \text{CACE}} + \color{#B7318A}{\pi_\text{A} \text{ATACE}} + \color{#FEBA2C}{\pi_\text{N} \text{NTACE}} \\[6pt] =\ & \color{#0D0887}{\pi_\text{C} \text{CACE}} + \color{#B7318A}{\pi_\text{A} \times 0} + \color{#FEBA2C}{\pi_\text{N} \times 0}\\[6pt] \text{ITT}\ =\ &\color{#0D0887}{\pi_\text{C} \text{CACE}} \end{aligned}$

26 / 50

$\color{#0D0887}{\text{CACE}} = \frac{\text{ITT}}{\color{#0D0887}{\pi_\text{C}}}$

26 / 50

$\color{#0D0887}{\text{CACE}} = \frac{\text{ITT}}{\color{#0D0887}{\pi_\text{C}}}$

ITT and π_C are both findable!

26 / 50

Finding the ITT

ITT = effect of assignment to treatment on outcome

$\text{ITT} = (\bar{y}\ |\ \text{Treatment}) - (\bar{y}\ |\ \text{Control)}$

27 / 50

Finding the ITT

ITT = effect of assignment to treatment on outcome

$\text{ITT} = (\bar{y}\ |\ \text{Treatment}) - (\bar{y}\ |\ \text{Control)}$

bed_nets %>% 
  group_by(treatment) %>% 
  summarize(avg = mean(health))

## # A tibble: 2 × 2
##   treatment   avg
##   <chr>     <dbl>
## 1 Control    40.9
## 2 Treatment  46.9

27 / 50

Finding the ITT

ITT = effect of assignment to treatment on outcome

$\text{ITT} = (\bar{y}\ |\ \text{Treatment}) - (\bar{y}\ |\ \text{Control)}$

bed_nets %>% 
  group_by(treatment) %>% 
  summarize(avg = mean(health))

## # A tibble: 2 × 2
##   treatment   avg
##   <chr>     <dbl>
## 1 Control    40.9
## 2 Treatment  46.9

itt_model <- lm(health ~ treatment, 
                data = bed_nets)
tidy(itt_model)

## # A tibble: 2 × 2
##   term               estimate
##   <chr>                 <dbl>
## 1 (Intercept)           40.9 
## 2 treatmentTreatment     5.99

27 / 50

Finding the π_C

People in treatment group who complied are a combination of Always Takers and Compliers

$\begin{aligned} \pi_\text{A} + \pi_\text{C} =& \text{% yes in treatment; or} \\ \pi_\text{C} =& \text{% yes in treatment} - \pi_\text{A} \end{aligned}$

28 / 50

Can we know π_A?

$\pi_\text{C} = \text{% yes in treatment} - \pi_\text{A}$

29 / 50

Can we know π_A?

$\pi_\text{C} = \text{% yes in treatment} - \pi_\text{A}$

We can assume that the proportion of Always Takers
is the same across treatment and control

29 / 50

Can we know π_A?

$\pi_\text{C} = \text{% yes in treatment} - \pi_\text{A}$

We can assume that the proportion of Always Takers
is the same across treatment and control

We know how many people were in control but still used nets—that's π_A!

29 / 50

Isolating π_C

$\begin{aligned} \pi_\text{C} =& \text{% yes in treatment} - \pi_\text{A} \\ =& \text{% yes in treatment} - \text{% yes in control} \end{aligned}$

bed_nets %>% 
  group_by(treatment, bed_net) %>% 
  summarize(n = n()) %>% 
  mutate(prop = n / sum(n))

## # A tibble: 4 × 4
## # Groups:   treatment [2]
##   treatment bed_net        n  prop
##   <chr>     <fct>      <int> <dbl>
## 1 Control   No bed net   808 0.805
## 2 Control   Bed net      196 0.195
## 3 Treatment No bed net   388 0.390
## 4 Treatment Bed net      608 0.610

30 / 50

Isolating π_C

$\begin{aligned} \pi_\text{C} =& \text{% yes in treatment} - \pi_\text{A} \\ =& \text{% yes in treatment} - \text{% yes in control} \end{aligned}$

bed_nets %>% 
  group_by(treatment, bed_net) %>% 
  summarize(n = n()) %>% 
  mutate(prop = n / sum(n))

## # A tibble: 4 × 4
## # Groups:   treatment [2]
##   treatment bed_net        n  prop
##   <chr>     <fct>      <int> <dbl>
## 1 Control   No bed net   808 0.805
## 2 Control   Bed net      196 0.195
## 3 Treatment No bed net   388 0.390
## 4 Treatment Bed net      608 0.610

# pi_c = prop yes in treatment - 
#        prop yes in control
pi_c <- 0.6104418 - 0.1952191
pi_c

## [1] 0.4152227

41.5% compliers!

30 / 50

Finding the CACE, finally!

$\text{CACE} = \frac{\text{ITT}}{{\pi_\text{C}}}$

31 / 50

Finding the CACE, finally!

$\text{CACE} = \frac{\text{ITT}}{{\pi_\text{C}}}$

ITT <- tidy(itt_model) %>% 
  filter(term == "treatmentTreatment") %>% 
  pull(estimate)
ITT

## [1] 5.987992

pi_c

## [1] 0.4152227

31 / 50

Finding the CACE, finally!

$\text{CACE} = \frac{\text{ITT}}{{\pi_\text{C}}}$

ITT <- tidy(itt_model) %>% 
  filter(term == "treatmentTreatment") %>% 
  pull(estimate)
ITT

## [1] 5.987992

pi_c

## [1] 0.4152227

CACE <- ITT / pi_c
CACE

## [1] 14.42116

Bed nets cause 14.4 point
increase in health for compliers

31 / 50

$\text{CACE} = \frac{\color{#FF4136}{\text{ITT}}}{\color{#B10DC9}{\pi_\text{C}}}$

$\color{#FF4136}{\text{ITT}} = (\bar{y}\ |\ \text{Treatment}) - (\bar{y}\ |\ \text{Control)}$

$\begin{aligned} \color{#B10DC9}{\pi_\text{C}}\ =\ & \text{% yes in treatment} - \\ & \text{% yes in control} \end{aligned}$

32 / 50

A faster way with 2SLS

LATE for the compliers

33 / 50

A faster way with 2SLS

LATE for the compliers

If you use assignment to treatment as an instrument,
you can find the causal effect for just compliers

33 / 50

A faster way with 2SLS

LATE for the compliers

If you use assignment to treatment as an instrument,
you can find the causal effect for just compliers

Instrumental variables in general give you the CACE

33 / 50

CACE with 2SLS

model_2sls <- iv_robust(health ~ bed_net | treatment, 
                        data = bed_nets)
tidy(model_2sls)

##             term estimate std.error statistic      p.value
## 1    (Intercept) 38.12285 0.5150818  74.01320 0.000000e+00
## 2 bed_netBed net 14.42116 1.2538198  11.50178 1.086989e-29

Same 14.421 effect!

34 / 50

Promotion
as an instrument35 / 50

Universal programs

What if you have a program
that anyone can opt in to?

36 / 50

Universal programs

What if you have a program
that anyone can opt in to?

ACA, voting, employer retirement matching

36 / 50

Universal programs

What if you have a program
that anyone can opt in to?

ACA, voting, employer retirement matching

You can't just look at outcomes
of participants vs. non-participants!

Selection bias!

36 / 50

Universal programs

What if you have a program
that anyone can opt in to?

ACA, voting, employer retirement matching

You can't just look at outcomes
of participants vs. non-participants!

Selection bias!

You can't randomly
assign people to it either

Ethics!

36 / 50

Randomized promotion

What if you encourage
some people to participate?

37 / 50

Randomized promotion

What if you encourage
some people to participate?

What if the encouragement is randomized?

37 / 50

Randomized promotion

What if you encourage
some people to participate?

What if the encouragement is randomized?

Valid treatment/control groups?

37 / 50

Randomized promotion

What if you encourage
some people to participate?

What if the encouragement is randomized?

Valid treatment/control groups?

Not really…

37 / 50

Randomized promotion38 / 50

Randomized promotion

…but also, kind of!

38 / 50

Randomized promotion

…but also, kind of!

Encouragement/promotion =
an instrument!

38 / 50

Not something weird? Does that work!?39 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Promotion causes people
to use the program. Yep.

39 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Promotion causes people
to use the program. Yep.

Exclusive?

Z → X → Y Z ↛ Y Cor(Z, Y | X) = 0

Promotion causes outcome
only through program? Yep.

39 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Promotion causes people
to use the program. Yep.

Exclusive?

Z → X → Y Z ↛ Y Cor(Z, Y | X) = 0

Promotion causes outcome
only through program? Yep.

Exogenous?

U ↛ Z Cor(Z, U) = 0

Unobserved things that influence outcome don't also influence promotion?
Yep.

39 / 50

Program compliance

Always Takers

People who will always
enroll in program

40 / 50

Program compliance

Always Takers

People who will always
enroll in program

Never Takers

People who will never
enroll in program

40 / 50

Program compliance

Always Takers

People who will always
enroll in program

Never Takers

People who will never
enroll in program

Compliers / Enrollers-if-Promoted

People who will enroll in the program if encouraged to

40 / 50

LATE for compliers
 
    id 
    outcome 
    program 
    promotion 
  


    1 
    45 
    TRUE 
    TRUE 
  

    2 
    55 
    TRUE 
    FALSE 
  

    3 
    52 
    FALSE 
    FALSE 
  

    4 
    39 
    FALSE 
    TRUE 
  


41 / 50

LATE for compliers
 
    id 
    outcome 
    program 
    promotion 
  


    1 
    45 
    TRUE 
    TRUE 
  

    2 
    55 
    TRUE 
    FALSE 
  

    3 
    52 
    FALSE 
    FALSE 
  

    4 
    39 
    FALSE 
    TRUE 
  


iv_robust(outcome ~ program | promotion)

41 / 50

LATE for compliers

id	outcome	program	promotion
1	45	TRUE	TRUE
2	55	TRUE	FALSE
3	52	FALSE	FALSE
4	39	FALSE	TRUE

iv_robust(outcome ~ program | promotion)

This will show the LATE for promoted-ees!

41 / 50

LATE for compliers

id	outcome	program	promotion
1	45	TRUE	TRUE
2	55	TRUE	FALSE
3	52	FALSE	FALSE
4	39	FALSE	TRUE

iv_robust(outcome ~ program | promotion)

This will show the LATE for promoted-ees!

Says nothing about the effect of the program on Always Takers or Never Takers

41 / 50

Fuzzy RDD42 / 50

Fuzzy discontinuities

43 / 50

Fuzzy discontinuities

Fuzzy discontinuities imply noncompliance

44 / 50

Fuzzy discontinuities

Fuzzy discontinuities imply noncompliance

Address noncompliance with
instrumental variables

44 / 50

What do we use as instrument?

Instrument = above/below cutoff

i.e. what they were supposed to do

45 / 50

What do we use as instrument?

Instrument = above/below cutoff

i.e. what they were supposed to do

(This is just like the CACE we just did!)

45 / 50

Not something weird? Does that work!?46 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Cutoff causes program? Yep.

46 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Cutoff causes program? Yep.

Exclusive?

Z → X → Y Z ↛ Y Cor(Z, Y | X) = 0

Cutoff causes outcome
only through program? Yep.

46 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Cutoff causes program? Yep.

Exclusive?

Z → X → Y Z ↛ Y Cor(Z, Y | X) = 0

Cutoff causes outcome
only through program? Yep.

Exogenous?

U ↛ Z Cor(Z, U) = 0

Unobserved things that influence outcome don't also influence cutoff?
It's an arbitrary cutoff, so sure.

46 / 50

Doubly local LATE

Effect is only for
(1) compliers (2) near the cutoff

Be specific when talking about effects;
definitely don't make population-level claims

47 / 50

Parametric fuzzy RD

Step 1: Center running variable + make threshold variable

tutoring_centered <- tutoring %>% 
  mutate(entrance_centered = entrance_exam - 70,
         below_cutoff = entrance_exam <= 70)
head(tutoring_centered, 6)

## # A tibble: 6 × 6
##      id entrance_exam tutoring exit_exam entrance_centered below_cutoff
##   <int>         <dbl> <lgl>        <dbl>             <dbl> <lgl>       
## 1     1          92.4 FALSE         78.1            22.4   FALSE       
## 2     2          72.8 FALSE         58.2             2.77  FALSE       
## 3     3          53.7 TRUE          62.0           -16.3   TRUE        
## 4     4          98.3 FALSE         67.5            28.3   FALSE       
## 5     5          69.7 TRUE          54.1            -0.288 TRUE        
## 6     6          68.1 TRUE          60.1            -1.93  TRUE

48 / 50

Parametric fuzzy RD

Step 2: Use cutoff as instrument in 2SLS model

# Bandwidth ± 10
fuzzy1 <- iv_robust(
  exit_exam ~ entrance_centered + tutoring | entrance_centered + below_cutoff,
  data = filter(tutoring_centered, entrance_centered >= -10 & entrance_centered <= 10)
)
tidy(fuzzy1)

##                term   estimate  std.error statistic       p.value
## 1       (Intercept) 60.1413558 1.01765573 59.097939 9.746624e-200
## 2 entrance_centered  0.4366281 0.09929619  4.397229  1.407213e-05
## 3      tutoringTRUE  9.7410444 1.91184891  5.095091  5.384163e-07

49 / 50

Nonparametric fuzzy RD

Use the fuzzy argument in rdrobust()

Important! Specify actual treatment status,
not the instrument of above/below the cutoff

rdrobust(y = tutoring$exit_exam, x = tutoring$entrance_exam, 
         c = 70, fuzzy = tutoring$tutoring) %>% 
  summary()

## First-stage estimates.
## 
## =============================================================================
##         Method     Coef. Std. Err.         z     P>|z|      [ 95% C.I. ]       
## =============================================================================
##   Conventional    -0.708     0.073    -9.751     0.000    [-0.850 , -0.565]

50 / 50

IV II &
RDD II

Session 12

PMAP 8521: Program evaluation
Andrew Young School of Policy Studies

1 / 50

Plan for today2 / 50

Plan for today

Treatment effects and compliance

2 / 50

Plan for today

Treatment effects and compliance

Randomized promotion

2 / 50

Plan for today

Treatment effects and compliance

Randomized promotion

Fuzzy regression discontinuity

2 / 50

Treatment effects
and compliance3 / 50

Potential outcomes

$\delta = (Y |\ P = 1) - (Y |\ P = 0)$

4 / 50

Potential outcomes

$\delta = (Y |\ P = 1) - (Y |\ P = 0)$

δ (delta) = causal effect

4 / 50

Potential outcomes

$\delta = (Y |\ P = 1) - (Y |\ P = 0)$

δ (delta) = causal effect

P = Program

4 / 50

Potential outcomes

$\delta = (Y |\ P = 1) - (Y |\ P = 0)$

δ (delta) = causal effect

P = Program

Y = Outcome

4 / 50

Potential outcomes

$\delta = (Y |\ P = 1) - (Y |\ P = 0)$

δ (delta) = causal effect

P = Program

Y = Outcome

$\delta = Y_1 - Y_0$

4 / 50

Fundamental problem
of causal inference

5 / 50

Fundamental problem
of causal inference

$\delta_i = Y_i^1 - Y_i^0 \quad \text{in real life is} \quad \delta_i = Y_i^1 - ???$

5 / 50

Fundamental problem
of causal inference

$\delta_i = Y_i^1 - Y_i^0 \quad \text{in real life is} \quad \delta_i = Y_i^1 - ???$

Individual-level effects are
impossible to observe!

5 / 50

Average treatment effect

Difference between average/expected value when
program is on vs. expected value when program is off

$\text{ATE} = E(Y_1 - Y_0) = E(Y_1) - E(Y_0)$

6 / 50

Average treatment effect

Difference between average/expected value when
program is on vs. expected value when program is off

$\text{ATE} = E(Y_1 - Y_0) = E(Y_1) - E(Y_0)$

Can be found for a whole population, on average

$\delta = (\bar{Y}\ |\ P = 1) - (\bar{Y}\ |\ P = 0)$

6 / 50

Every individual has a
treatment/causal effect

7 / 50

Every individual has a
treatment/causal effect

ATE = average of all
unit-level causal effects

7 / 50

Every individual has a
treatment/causal effect

ATE = average of all
unit-level causal effects

ATE = Average effect
for the whole population

7 / 50

Other versions of causal effects

Average treatment on the treated

ATT/TOT

8 / 50

Other versions of causal effects

Average treatment on the treated

ATT/TOT

Conditional average treatment effect

CATE

8 / 50

Local effects

9 / 50

LATE

Local average treatment effect (LATE) =
weighted ATE

Narrower effect; only applies to some of the population

10 / 50

LATE

Local average treatment effect (LATE) =
weighted ATE

Narrower effect; only applies to some of the population

You can't make population-level
claims with LATE

10 / 50

LATE

Local average treatment effect (LATE) =
weighted ATE

Narrower effect; only applies to some of the population

You can't make population-level
claims with LATE

(But that can be okay!)

10 / 50

LATE

11 / 50

LATE

In RDD, LATE = people in the bandwidth

11 / 50

LATE

In RDD, LATE = people in the bandwidth

In RCTs and IVs, LATE = compliers

11 / 50

Compliance

Complier

Treatment
follows assignment

Always taker

Gets treatment
regardless of assignment

12 / 50

Compliance

Complier

Treatment
follows assignment

Always taker

Gets treatment
regardless of assignment

Never taker

Rejects treatment
regardless of assignment

Defier

Does the opposite
of assignment

12 / 50

13 / 50

Ignoring defiers14 / 50

Ignoring defiers

We can generally assume
that defiers don't exist

14 / 50

Ignoring defiers

We can generally assume
that defiers don't exist

In drug trials this makes sense; you can't get access
to medicine without being in treatment group

14 / 50

Ignoring defiers

We can generally assume
that defiers don't exist

In drug trials this makes sense; you can't get access
to medicine without being in treatment group

In development it can make sense; in a bed net RCT,
a defier assigned to treatment would have to
tear down all existing bed nets out of spite

14 / 50

Ignoring defiers

15 / 50

Ignoring defiers

Monotonicity assumption

15 / 50

Ignoring defiers

Monotonicity assumption

Assignment to treatment only
has an effect in one direction

15 / 50

Ignoring defiers

Monotonicity assumption

Assignment to treatment only
has an effect in one direction

Assignment to treatment can only
increase—not decrease—your actual chance of treatment

15 / 50

16 / 50

More causal effects

Intent to treat (ITT)

Effect of assignment (not actual treatment!)

17 / 50

More causal effects

Intent to treat (ITT)

Effect of assignment (not actual treatment!)

17 / 50

More causal effects

Complier Average Causal Effect (CACE)

LATE for the compliers

18 / 50

More causal effects

Complier Average Causal Effect (CACE)

LATE for the compliers

18 / 50

Hypothetical bed net program

An NGO distributes mosquito bed nets to help
improve health by reducing malaria infection rate

19 / 50

Hypothetical bed net program

An NGO distributes mosquito bed nets to help
improve health by reducing malaria infection rate

We can read everyone's minds and we know if
people are always takers, never takers, or compliers

19 / 50

Mind reading

20 / 50

Actual data

But we can't read minds! This is what we actually see:

21 / 50

Actual data

(Actually this is what we see)

22 / 50

23 / 50

24 / 50

$\text{ITT}\ =\ \color{#0D0887}{\pi_\text{C} \text{CACE}} + \color{#B7318A}{\pi_\text{A} \text{ATACE}} + \color{#FEBA2C}{\pi_\text{N} \text{NTACE}}$

25 / 50

$\text{ITT}\ =\ \color{#0D0887}{\pi_\text{C} \text{CACE}} + \color{#B7318A}{\pi_\text{A} \times 0} + \color{#FEBA2C}{\pi_\text{N} \times 0}$

25 / 50

26 / 50

$\color{#0D0887}{\text{CACE}} = \frac{\text{ITT}}{\color{#0D0887}{\pi_\text{C}}}$

26 / 50

$\color{#0D0887}{\text{CACE}} = \frac{\text{ITT}}{\color{#0D0887}{\pi_\text{C}}}$

ITT and π_C are both findable!

26 / 50

Finding the ITT

ITT = effect of assignment to treatment on outcome

$\text{ITT} = (\bar{y}\ |\ \text{Treatment}) - (\bar{y}\ |\ \text{Control)}$

27 / 50

Finding the ITT

ITT = effect of assignment to treatment on outcome

$\text{ITT} = (\bar{y}\ |\ \text{Treatment}) - (\bar{y}\ |\ \text{Control)}$

bed_nets %>% 
  group_by(treatment) %>% 
  summarize(avg = mean(health))

## # A tibble: 2 × 2
##   treatment   avg
##   <chr>     <dbl>
## 1 Control    40.9
## 2 Treatment  46.9

27 / 50

Finding the ITT

ITT = effect of assignment to treatment on outcome

$\text{ITT} = (\bar{y}\ |\ \text{Treatment}) - (\bar{y}\ |\ \text{Control)}$

bed_nets %>% 
  group_by(treatment) %>% 
  summarize(avg = mean(health))

## # A tibble: 2 × 2
##   treatment   avg
##   <chr>     <dbl>
## 1 Control    40.9
## 2 Treatment  46.9

itt_model <- lm(health ~ treatment, 
                data = bed_nets)
tidy(itt_model)

## # A tibble: 2 × 2
##   term               estimate
##   <chr>                 <dbl>
## 1 (Intercept)           40.9 
## 2 treatmentTreatment     5.99

27 / 50

Finding the π_C

People in treatment group who complied are a combination of Always Takers and Compliers

$\begin{aligned} \pi_\text{A} + \pi_\text{C} =& \text{% yes in treatment; or} \\ \pi_\text{C} =& \text{% yes in treatment} - \pi_\text{A} \end{aligned}$

28 / 50

Can we know π_A?

$\pi_\text{C} = \text{% yes in treatment} - \pi_\text{A}$

29 / 50

Can we know π_A?

$\pi_\text{C} = \text{% yes in treatment} - \pi_\text{A}$

We can assume that the proportion of Always Takers
is the same across treatment and control

29 / 50

Can we know π_A?

$\pi_\text{C} = \text{% yes in treatment} - \pi_\text{A}$

We can assume that the proportion of Always Takers
is the same across treatment and control

We know how many people were in control but still used nets—that's π_A!

29 / 50

Isolating π_C

$\begin{aligned} \pi_\text{C} =& \text{% yes in treatment} - \pi_\text{A} \\ =& \text{% yes in treatment} - \text{% yes in control} \end{aligned}$

bed_nets %>% 
  group_by(treatment, bed_net) %>% 
  summarize(n = n()) %>% 
  mutate(prop = n / sum(n))

## # A tibble: 4 × 4
## # Groups:   treatment [2]
##   treatment bed_net        n  prop
##   <chr>     <fct>      <int> <dbl>
## 1 Control   No bed net   808 0.805
## 2 Control   Bed net      196 0.195
## 3 Treatment No bed net   388 0.390
## 4 Treatment Bed net      608 0.610

30 / 50

Isolating π_C

$\begin{aligned} \pi_\text{C} =& \text{% yes in treatment} - \pi_\text{A} \\ =& \text{% yes in treatment} - \text{% yes in control} \end{aligned}$

bed_nets %>% 
  group_by(treatment, bed_net) %>% 
  summarize(n = n()) %>% 
  mutate(prop = n / sum(n))

## # A tibble: 4 × 4
## # Groups:   treatment [2]
##   treatment bed_net        n  prop
##   <chr>     <fct>      <int> <dbl>
## 1 Control   No bed net   808 0.805
## 2 Control   Bed net      196 0.195
## 3 Treatment No bed net   388 0.390
## 4 Treatment Bed net      608 0.610

# pi_c = prop yes in treatment - 
#        prop yes in control
pi_c <- 0.6104418 - 0.1952191
pi_c

## [1] 0.4152227

41.5% compliers!

30 / 50

Finding the CACE, finally!

$\text{CACE} = \frac{\text{ITT}}{{\pi_\text{C}}}$

31 / 50

Finding the CACE, finally!

$\text{CACE} = \frac{\text{ITT}}{{\pi_\text{C}}}$

ITT <- tidy(itt_model) %>% 
  filter(term == "treatmentTreatment") %>% 
  pull(estimate)
ITT

## [1] 5.987992

pi_c

## [1] 0.4152227

31 / 50

Finding the CACE, finally!

$\text{CACE} = \frac{\text{ITT}}{{\pi_\text{C}}}$

ITT <- tidy(itt_model) %>% 
  filter(term == "treatmentTreatment") %>% 
  pull(estimate)
ITT

## [1] 5.987992

pi_c

## [1] 0.4152227

CACE <- ITT / pi_c
CACE

## [1] 14.42116

Bed nets cause 14.4 point
increase in health for compliers

31 / 50

$\text{CACE} = \frac{\color{#FF4136}{\text{ITT}}}{\color{#B10DC9}{\pi_\text{C}}}$

$\color{#FF4136}{\text{ITT}} = (\bar{y}\ |\ \text{Treatment}) - (\bar{y}\ |\ \text{Control)}$

$\begin{aligned} \color{#B10DC9}{\pi_\text{C}}\ =\ & \text{% yes in treatment} - \\ & \text{% yes in control} \end{aligned}$

32 / 50

A faster way with 2SLS

LATE for the compliers

33 / 50

A faster way with 2SLS

LATE for the compliers

If you use assignment to treatment as an instrument,
you can find the causal effect for just compliers

33 / 50

A faster way with 2SLS

LATE for the compliers

If you use assignment to treatment as an instrument,
you can find the causal effect for just compliers

Instrumental variables in general give you the CACE

33 / 50

CACE with 2SLS

model_2sls <- iv_robust(health ~ bed_net | treatment, 
                        data = bed_nets)
tidy(model_2sls)

##             term estimate std.error statistic      p.value
## 1    (Intercept) 38.12285 0.5150818  74.01320 0.000000e+00
## 2 bed_netBed net 14.42116 1.2538198  11.50178 1.086989e-29

Same 14.421 effect!

34 / 50

Promotion
as an instrument35 / 50

Universal programs

What if you have a program
that anyone can opt in to?

36 / 50

Universal programs

What if you have a program
that anyone can opt in to?

ACA, voting, employer retirement matching

36 / 50

Universal programs

What if you have a program
that anyone can opt in to?

ACA, voting, employer retirement matching

You can't just look at outcomes
of participants vs. non-participants!

Selection bias!

36 / 50

Universal programs

What if you have a program
that anyone can opt in to?

ACA, voting, employer retirement matching

You can't just look at outcomes
of participants vs. non-participants!

Selection bias!

You can't randomly
assign people to it either

Ethics!

36 / 50

Randomized promotion

What if you encourage
some people to participate?

37 / 50

Randomized promotion

What if you encourage
some people to participate?

What if the encouragement is randomized?

37 / 50

Randomized promotion

What if you encourage
some people to participate?

What if the encouragement is randomized?

Valid treatment/control groups?

37 / 50

Randomized promotion

What if you encourage
some people to participate?

What if the encouragement is randomized?

Valid treatment/control groups?

Not really…

37 / 50

Randomized promotion38 / 50

Randomized promotion

…but also, kind of!

38 / 50

Randomized promotion

…but also, kind of!

Encouragement/promotion =
an instrument!

38 / 50

Not something weird? Does that work!?39 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Promotion causes people
to use the program. Yep.

39 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Promotion causes people
to use the program. Yep.

Exclusive?

Z → X → Y Z ↛ Y Cor(Z, Y | X) = 0

Promotion causes outcome
only through program? Yep.

39 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Promotion causes people
to use the program. Yep.

Exclusive?

Z → X → Y Z ↛ Y Cor(Z, Y | X) = 0

Promotion causes outcome
only through program? Yep.

Exogenous?

U ↛ Z Cor(Z, U) = 0

Unobserved things that influence outcome don't also influence promotion?
Yep.

39 / 50

Program compliance

Always Takers

People who will always
enroll in program

40 / 50

Program compliance

Always Takers

People who will always
enroll in program

Never Takers

People who will never
enroll in program

40 / 50

Program compliance

Always Takers

People who will always
enroll in program

Never Takers

People who will never
enroll in program

Compliers / Enrollers-if-Promoted

People who will enroll in the program if encouraged to

40 / 50

LATE for compliers
 
    id 
    outcome 
    program 
    promotion 
  


    1 
    45 
    TRUE 
    TRUE 
  

    2 
    55 
    TRUE 
    FALSE 
  

    3 
    52 
    FALSE 
    FALSE 
  

    4 
    39 
    FALSE 
    TRUE 
  


41 / 50

LATE for compliers
 
    id 
    outcome 
    program 
    promotion 
  


    1 
    45 
    TRUE 
    TRUE 
  

    2 
    55 
    TRUE 
    FALSE 
  

    3 
    52 
    FALSE 
    FALSE 
  

    4 
    39 
    FALSE 
    TRUE 
  


iv_robust(outcome ~ program | promotion)

41 / 50

LATE for compliers

id	outcome	program	promotion
1	45	TRUE	TRUE
2	55	TRUE	FALSE
3	52	FALSE	FALSE
4	39	FALSE	TRUE

iv_robust(outcome ~ program | promotion)

This will show the LATE for promoted-ees!

41 / 50

LATE for compliers

id	outcome	program	promotion
1	45	TRUE	TRUE
2	55	TRUE	FALSE
3	52	FALSE	FALSE
4	39	FALSE	TRUE

iv_robust(outcome ~ program | promotion)

This will show the LATE for promoted-ees!

Says nothing about the effect of the program on Always Takers or Never Takers

41 / 50

Fuzzy RDD42 / 50

Fuzzy discontinuities

43 / 50

Fuzzy discontinuities

Fuzzy discontinuities imply noncompliance

44 / 50

Fuzzy discontinuities

Fuzzy discontinuities imply noncompliance

Address noncompliance with
instrumental variables

44 / 50

What do we use as instrument?

Instrument = above/below cutoff

i.e. what they were supposed to do

45 / 50

What do we use as instrument?

Instrument = above/below cutoff

i.e. what they were supposed to do

(This is just like the CACE we just did!)

45 / 50

Not something weird? Does that work!?46 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Cutoff causes program? Yep.

46 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Cutoff causes program? Yep.

Exclusive?

Z → X → Y Z ↛ Y Cor(Z, Y | X) = 0

Cutoff causes outcome
only through program? Yep.

46 / 50

Not something weird? Does that work!?

Relevant?

Z → X Cor(Z, X) ≠ 0

Cutoff causes program? Yep.

Exclusive?

Z → X → Y Z ↛ Y Cor(Z, Y | X) = 0

Cutoff causes outcome
only through program? Yep.

Exogenous?

U ↛ Z Cor(Z, U) = 0

Unobserved things that influence outcome don't also influence cutoff?
It's an arbitrary cutoff, so sure.

46 / 50

Doubly local LATE

Effect is only for
(1) compliers (2) near the cutoff

Be specific when talking about effects;
definitely don't make population-level claims

47 / 50

Parametric fuzzy RD

Step 1: Center running variable + make threshold variable

tutoring_centered <- tutoring %>% 
  mutate(entrance_centered = entrance_exam - 70,
         below_cutoff = entrance_exam <= 70)
head(tutoring_centered, 6)

## # A tibble: 6 × 6
##      id entrance_exam tutoring exit_exam entrance_centered below_cutoff
##   <int>         <dbl> <lgl>        <dbl>             <dbl> <lgl>       
## 1     1          92.4 FALSE         78.1            22.4   FALSE       
## 2     2          72.8 FALSE         58.2             2.77  FALSE       
## 3     3          53.7 TRUE          62.0           -16.3   TRUE        
## 4     4          98.3 FALSE         67.5            28.3   FALSE       
## 5     5          69.7 TRUE          54.1            -0.288 TRUE        
## 6     6          68.1 TRUE          60.1            -1.93  TRUE

48 / 50

Parametric fuzzy RD

Step 2: Use cutoff as instrument in 2SLS model

# Bandwidth ± 10
fuzzy1 <- iv_robust(
  exit_exam ~ entrance_centered + tutoring | entrance_centered + below_cutoff,
  data = filter(tutoring_centered, entrance_centered >= -10 & entrance_centered <= 10)
)
tidy(fuzzy1)

##                term   estimate  std.error statistic       p.value
## 1       (Intercept) 60.1413558 1.01765573 59.097939 9.746624e-200
## 2 entrance_centered  0.4366281 0.09929619  4.397229  1.407213e-05
## 3      tutoringTRUE  9.7410444 1.91184891  5.095091  5.384163e-07

49 / 50

Nonparametric fuzzy RD

Use the fuzzy argument in rdrobust()

Important! Specify actual treatment status,
not the instrument of above/below the cutoff

rdrobust(y = tutoring$exit_exam, x = tutoring$entrance_exam, 
         c = 70, fuzzy = tutoring$tutoring) %>% 
  summary()

## First-stage estimates.
## 
## =============================================================================
##         Method     Coef. Std. Err.         z     P>|z|      [ 95% C.I. ]       
## =============================================================================
##   Conventional    -0.708     0.073    -9.751     0.000    [-0.850 , -0.565]

50 / 50

↑, ←, Pg Up, k	Go to previous slide
↓, →, Pg Dn, Space, j	Go to next slide
Home	Go to first slide
End	Go to last slide
Number + Return	Go to specific slide
b / m / f	Toggle blackout / mirrored / fullscreen mode
c	Clone slideshow
p	Toggle presenter mode
t	Restart the presentation timer
?, h	Toggle this help
o	Tile View: Overview of Slides

IV II &RDD II

Plan for today

Plan for today

Plan for today

Plan for today

Treatment effectsand compliance

Potential outcomes

Potential outcomes

Potential outcomes

Potential outcomes

Potential outcomes

Average treatment effect

Average treatment effect

Other versions of causal effects

Other versions of causal effects

Local effects

LATE

LATE

LATE

LATE

LATE

LATE

Compliance

Compliance

Ignoring defiers

Ignoring defiers

Ignoring defiers

Ignoring defiers

Ignoring defiers

Ignoring defiers

Ignoring defiers

Ignoring defiers

More causal effects

More causal effects

More causal effects

More causal effects

Hypothetical bed net program

Hypothetical bed net program

Mind reading

Actual data

Actual data

Finding the ITT

Finding the ITT

Finding the ITT

Finding the πC

Can we know πA?

Can we know πA?

Can we know πA?

Isolating πC

Isolating πC

Finding the CACE, finally!

Finding the CACE, finally!

Finding the CACE, finally!

A faster way with 2SLS

A faster way with 2SLS

A faster way with 2SLS

CACE with 2SLS

Promotionas an instrument

Universal programs

Universal programs

Universal programs

Universal programs

Randomized promotion

Randomized promotion

Randomized promotion

Randomized promotion

Randomized promotion

Randomized promotion

Randomized promotion

Not something weird? Does that work!?

Not something weird? Does that work!?

Not something weird? Does that work!?

Not something weird? Does that work!?

Program compliance

Program compliance

Program compliance

LATE for compliers

LATE for compliers

LATE for compliers

LATE for compliers

IV II &
RDD II

Treatment effects
and compliance

Finding the π_C

Can we know π_A?

Can we know π_A?

Can we know π_A?

Isolating π_C

Isolating π_C

Promotion
as an instrument

IV II &
RDD II

Treatment effects
and compliance

Finding the π_C

Can we know π_A?

Can we know π_A?

Can we know π_A?

Isolating π_C

Isolating π_C