In-person
session 5

September 19, 2022

PMAP 8521: Program evaluation
Andrew Young School of Policy Studies

1 / 46

Plan for today2 / 46

Plan for today

DAGs, continued

2 / 46

Plan for today

DAGs, continued

Potential outcomes vs. do() notation

2 / 46

Plan for today

DAGs, continued

Potential outcomes vs. do() notation

do-calculus, adjustment, and CATEs

2 / 46

Plan for today

DAGs, continued

Potential outcomes vs. do() notation

do-calculus, adjustment, and CATEs

Logic models, DAGs, and measurement

2 / 46

DAGs, continued3 / 46

4 / 46

5 / 46

6 / 46

Effect of race on police use of force
using administrative data

7 / 46

Effect of race on police use of force
using administrative data

8 / 46

Smoking → Cardiac arrest example

9 / 46

Person	Smoker	Cardiac arrest	Cholesterol	Weight	Lifestyle healthiness
1	TRUE	TRUE	150	170	6
2	TRUE	FALSE	170	180	3
3	FALSE	FALSE	130	110	9
4	FALSE	TRUE	140	140	8
5	TRUE	TRUE	120	150	2
6	TRUE	FALSE	130	230	3
7	FALSE	FALSE	140	250	10

dag {
bb="0,0,1,1"
"Cardiac arrest" [outcome,pos="0.599,0.432"]
Cholesterol [pos="0.415,0.440"]
Lifestyle [pos="0.156,0.317"]
Smoking [exposure,pos="0.243,0.428"]
Weight [adjusted,pos="0.297,0.255"]
Cholesterol -> "Cardiac arrest"
Lifestyle -> Smoking
Lifestyle -> Weight
Smoking -> Cholesterol
Weight -> Cholesterol
}

How can you be sure
you include everything in a DAG?

How do you know when to stop?

Is there a rule of thumb
for the number of nodes?

10 / 46

Why can we combine nodes in a DAG if they
don't represent the same concept?

Why include unmeasurable things in a DAG?

11 / 46

Why do DAGs have to be acyclic?

What if there really is reverse causation?

12 / 46

How do we actually
adjust for these things?

13 / 46

Potential outcomes
vs. do() notation14 / 46

Expectations

$\operatorname{E}(\cdot), \mathbf{E}(\cdot), \mathbb{E}(\cdot) \quad \text{vs.}\quad \operatorname{P}(\cdot)$

Basically a fancy way of saying "average"

15 / 46

Outcomes and programs

16 / 46

Causal effects with potential outcomes

$\begin{aligned} & \textbf{Potential outcomes notation:} \\ \delta\ =&\ {\textstyle \frac{1}{n} \sum_{i=1}^n} Y_i (1) - Y_i (0) \\ & \\ & \text{or alternatively with } \textbf{E} \\ \delta\ =&\ \textbf{E} [Y_i (1) - Y_i (0)] \\ \end{aligned}$

17 / 46

Causal effects with do()

$\begin{aligned} & \textbf{Pearl notation:} \\ \delta\ =&\ \textbf{E}[Y_i \mid \operatorname{do}(X = 1) - Y_i \mid \operatorname{do}(X = 0)] \\ & \\ & \text{or more simply} \\ \delta\ =&\ \textbf{E} [Y_i \mid \operatorname{do}(X)] \\ \end{aligned}$

18 / 46

$\begin{aligned} \textbf{E} [Y_i\ \mid\ &\operatorname{do}(X)] \quad \\ &= \\ \quad \textbf{E} [Y_i (1&) - Y_i (0)] \end{aligned}$

19 / 46

We can't see this

$\textbf{E} [Y_i \mid \operatorname{do}(X)] \quad \text{or} \quad \textbf{E} [Y_i (1) - Y_i (0)]$

So we find the average causal effect (ACE)

$\hat{\delta} = \textbf{E} [Y_i \mid X = 1] - \textbf{E} [Y_i \mid X = 0]$

20 / 46

21 / 46

do-calculus,
adjustment, and CATEs22 / 46

DAGs and identification23 / 46

DAGs and identification

DAGs are a statistical tool, but they don't
tell you what statistical method to use

23 / 46

DAGs and identification

DAGs are a statistical tool, but they don't
tell you what statistical method to use

DAGs help you with the identification strategy

23 / 46

24 / 46

https://twitter.com/RepThomasMassie/status/1491441851748204546

Easist identification

Identification through research design

RCTs

When treatment is randomized, delete all arrows going into it

No need for any do-calculus!

25 / 46

Most other identification

Identification through do-calculus

Rules for graph surgery

Backdoor adjustment and frontdoor adjustment
are special common patterns of do-calculus

26 / 46

Where can we learn more about do-calculus?

Here!

27 / 46

Rule 1: Decide if we can ignore an observation

$P(y \mid z, \operatorname{do}(x), w) = P(y \mid \operatorname{do}(x), w) \qquad \text{ if } (Y \perp Z \mid W, X)_{G_{\overline{X}}}$

Rule 2: Decide if we can treat an intervention as an observation

$P(y \mid \operatorname{do}(z), \operatorname{do}(x), w) = P(y \mid z, \operatorname{do}(x), w) \qquad \text{ if } (Y \perp Z \mid W, X)_{G_{\overline{X}, \underline{Z}}}$

Rule 3: Decide if we can ignore an intervention

$P(y \mid \operatorname{do}(z), \operatorname{do}(x), w) = P(y \mid \operatorname{do}(x), w) \qquad \text{ if } (Y \perp Z \mid W, X)_{G_{\overline{X}, \overline{Z(W)}}}$

28 / 46

29 / 46

Adjusting for backdoor confounding

30 / 46

Adjusting for frontdoor confounding

31 / 46

Smoking/tar + Uber

Effect of shared rides on tips; use frontdoor magic

Like IV but in reverse:

IV: instrument → treatment → outcome
Frontdoor: treatment → instrumenty-mediator → outcome

dag {
bb="0,0,1,1"
"Actually take shared ride" [pos="0.528,0.508"]
"Authorize shared ride" [exposure,pos="0.288,0.504"]
"Lots of unobserved stuff" [pos="0.521,0.342"]
"Tip driver" [outcome,pos="0.743,0.518"]
"Actually take shared ride" -> "Tip driver"
"Authorize shared ride" -> "Actually take shared ride"
"Lots of unobserved stuff" -> "Authorize shared ride"
"Lots of unobserved stuff" -> "Tip driver"
}

https://twitter.com/andrewheiss/status/1361686426820222977

More complex DAGs without
obvious backdoor or frontdoor solutions

Chug through the rules of do-calculus
to see if the relationship is identifiable

Causal Fusion

32 / 46

33 / 46

34 / 46

35 / 46

36 / 46

When things are identified, there are
still arrows leading into Y.
What do we do with those?
How do you explain those relationships?

37 / 46

When things are identified, there are
still arrows leading into Y.
What do we do with those?
How do you explain those relationships?

Outcomes have multiple causes.
How do you justify that your proposed
cause is the most causal factor?

37 / 46

100% depends on your research question

Why can't we just subtract the averages
between treated and untreated groups?

38 / 46

When you're making groups for CATE, how do
you decide what groups to put people in?

Slides from lecture

39 / 46

Unconfoundedness assumption

How can we assume/pretend that treatment was
randomly assigned within each age?

It seems unlikely. Wouldn't there be other factors within the older/younger group that make a person more/less likely to engage in treatment (e.g., health status)?

Slides from lecture

40 / 46

Does every research question
need an identification strategy?

41 / 46

Does every research question
need an identification strategy?

No!

Correlation alone is okay!
Can lead to more focused causal questions later!

41 / 46

42 / 46

A correlational study found that MS was strongly associated with Epstein-Barr virus (EBV) - they don't know the exact mechanism yet, but because of mRNA vaccine technology, they can develop vaccines against EBV and help stop MS. They'll figure out exact mechanisms later. For now, they've started clinical trials.

https://www.forbes.com/sites/roberthart/2022/01/14/moderna-starts-human-trials-of-mrna-vaccine-for-virus-that-likely-causes-multiple-sclerosis/?sh=74f52ca51a04

Logic models, DAGs,
and measurement43 / 46

What's the difference between
logic models and DAGs?

Can't I just remake my logic model in Dagitty and be done?

44 / 46

DAGs vs. Logic models

DAGs are a statistical tool

Describe a data-generating process
and isolate/identify relationships

45 / 46

DAGs vs. Logic models

DAGs are a statistical tool

Describe a data-generating process
and isolate/identify relationships

Logic models are a managerial tool

Oversee the inner workings of a program and its theory

45 / 46

46 / 46

https://theconversation.com/how-cities-can-add-accessible-green-space-in-a-post-coronavirus-world-139194

https://sf.eater.com/2020/5/14/21258980/berkeley-coronavirus-covid-19-jesse-arreguin-street-closures

In-person
session 5

September 19, 2022

PMAP 8521: Program evaluation
Andrew Young School of Policy Studies

1 / 46

Plan for today2 / 46

Plan for today

DAGs, continued

2 / 46

Plan for today

DAGs, continued

Potential outcomes vs. do() notation

2 / 46

Plan for today

DAGs, continued

Potential outcomes vs. do() notation

do-calculus, adjustment, and CATEs

2 / 46

Plan for today

DAGs, continued

Potential outcomes vs. do() notation

do-calculus, adjustment, and CATEs

Logic models, DAGs, and measurement

2 / 46

DAGs, continued3 / 46

4 / 46

5 / 46

6 / 46

Effect of race on police use of force
using administrative data

7 / 46

Effect of race on police use of force
using administrative data

8 / 46

Smoking → Cardiac arrest example

9 / 46

Person	Smoker	Cardiac arrest	Cholesterol	Weight	Lifestyle healthiness
1	TRUE	TRUE	150	170	6
2	TRUE	FALSE	170	180	3
3	FALSE	FALSE	130	110	9
4	FALSE	TRUE	140	140	8
5	TRUE	TRUE	120	150	2
6	TRUE	FALSE	130	230	3
7	FALSE	FALSE	140	250	10

dag {
bb="0,0,1,1"
"Cardiac arrest" [outcome,pos="0.599,0.432"]
Cholesterol [pos="0.415,0.440"]
Lifestyle [pos="0.156,0.317"]
Smoking [exposure,pos="0.243,0.428"]
Weight [adjusted,pos="0.297,0.255"]
Cholesterol -> "Cardiac arrest"
Lifestyle -> Smoking
Lifestyle -> Weight
Smoking -> Cholesterol
Weight -> Cholesterol
}

How can you be sure
you include everything in a DAG?

How do you know when to stop?

Is there a rule of thumb
for the number of nodes?

10 / 46

Why can we combine nodes in a DAG if they
don't represent the same concept?

Why include unmeasurable things in a DAG?

11 / 46

Why do DAGs have to be acyclic?

What if there really is reverse causation?

12 / 46

How do we actually
adjust for these things?

13 / 46

Potential outcomes
vs. do() notation14 / 46

Expectations

$\operatorname{E}(\cdot), \mathbf{E}(\cdot), \mathbb{E}(\cdot) \quad \text{vs.}\quad \operatorname{P}(\cdot)$

Basically a fancy way of saying "average"

15 / 46

Outcomes and programs

16 / 46

Causal effects with potential outcomes

17 / 46

Causal effects with do()

18 / 46

$\begin{aligned} \textbf{E} [Y_i\ \mid\ &\operatorname{do}(X)] \quad \\ &= \\ \quad \textbf{E} [Y_i (1&) - Y_i (0)] \end{aligned}$

19 / 46

We can't see this

$\textbf{E} [Y_i \mid \operatorname{do}(X)] \quad \text{or} \quad \textbf{E} [Y_i (1) - Y_i (0)]$

So we find the average causal effect (ACE)

$\hat{\delta} = \textbf{E} [Y_i \mid X = 1] - \textbf{E} [Y_i \mid X = 0]$

20 / 46

21 / 46

do-calculus,
adjustment, and CATEs22 / 46

DAGs and identification23 / 46

DAGs and identification

DAGs are a statistical tool, but they don't
tell you what statistical method to use

23 / 46

DAGs and identification

DAGs are a statistical tool, but they don't
tell you what statistical method to use

DAGs help you with the identification strategy

23 / 46

24 / 46

https://twitter.com/RepThomasMassie/status/1491441851748204546

Easist identification

Identification through research design

RCTs

When treatment is randomized, delete all arrows going into it

No need for any do-calculus!

25 / 46

Most other identification

Identification through do-calculus

Rules for graph surgery

Backdoor adjustment and frontdoor adjustment
are special common patterns of do-calculus

26 / 46

Where can we learn more about do-calculus?

Here!

27 / 46

Rule 1: Decide if we can ignore an observation

$P(y \mid z, \operatorname{do}(x), w) = P(y \mid \operatorname{do}(x), w) \qquad \text{ if } (Y \perp Z \mid W, X)_{G_{\overline{X}}}$

Rule 2: Decide if we can treat an intervention as an observation

$P(y \mid \operatorname{do}(z), \operatorname{do}(x), w) = P(y \mid z, \operatorname{do}(x), w) \qquad \text{ if } (Y \perp Z \mid W, X)_{G_{\overline{X}, \underline{Z}}}$

Rule 3: Decide if we can ignore an intervention

$P(y \mid \operatorname{do}(z), \operatorname{do}(x), w) = P(y \mid \operatorname{do}(x), w) \qquad \text{ if } (Y \perp Z \mid W, X)_{G_{\overline{X}, \overline{Z(W)}}}$

28 / 46

29 / 46

Adjusting for backdoor confounding

30 / 46

Adjusting for frontdoor confounding

31 / 46

Smoking/tar + Uber

Effect of shared rides on tips; use frontdoor magic

Like IV but in reverse:

IV: instrument → treatment → outcome
Frontdoor: treatment → instrumenty-mediator → outcome

dag {
bb="0,0,1,1"
"Actually take shared ride" [pos="0.528,0.508"]
"Authorize shared ride" [exposure,pos="0.288,0.504"]
"Lots of unobserved stuff" [pos="0.521,0.342"]
"Tip driver" [outcome,pos="0.743,0.518"]
"Actually take shared ride" -> "Tip driver"
"Authorize shared ride" -> "Actually take shared ride"
"Lots of unobserved stuff" -> "Authorize shared ride"
"Lots of unobserved stuff" -> "Tip driver"
}

https://twitter.com/andrewheiss/status/1361686426820222977

More complex DAGs without
obvious backdoor or frontdoor solutions

Chug through the rules of do-calculus
to see if the relationship is identifiable

Causal Fusion

32 / 46

33 / 46

34 / 46

35 / 46

36 / 46

When things are identified, there are
still arrows leading into Y.
What do we do with those?
How do you explain those relationships?

37 / 46

When things are identified, there are
still arrows leading into Y.
What do we do with those?
How do you explain those relationships?

Outcomes have multiple causes.
How do you justify that your proposed
cause is the most causal factor?

37 / 46

100% depends on your research question

Why can't we just subtract the averages
between treated and untreated groups?

38 / 46

When you're making groups for CATE, how do
you decide what groups to put people in?

Slides from lecture

39 / 46

Unconfoundedness assumption

How can we assume/pretend that treatment was
randomly assigned within each age?

It seems unlikely. Wouldn't there be other factors within the older/younger group that make a person more/less likely to engage in treatment (e.g., health status)?

Slides from lecture

40 / 46

Does every research question
need an identification strategy?

41 / 46

Does every research question
need an identification strategy?

No!

Correlation alone is okay!
Can lead to more focused causal questions later!

41 / 46

42 / 46

https://www.forbes.com/sites/roberthart/2022/01/14/moderna-starts-human-trials-of-mrna-vaccine-for-virus-that-likely-causes-multiple-sclerosis/?sh=74f52ca51a04

Logic models, DAGs,
and measurement43 / 46

What's the difference between
logic models and DAGs?

Can't I just remake my logic model in Dagitty and be done?

44 / 46

DAGs vs. Logic models

DAGs are a statistical tool

Describe a data-generating process
and isolate/identify relationships

45 / 46

DAGs vs. Logic models

DAGs are a statistical tool

Describe a data-generating process
and isolate/identify relationships

Logic models are a managerial tool

Oversee the inner workings of a program and its theory

45 / 46

46 / 46

https://theconversation.com/how-cities-can-add-accessible-green-space-in-a-post-coronavirus-world-139194

https://sf.eater.com/2020/5/14/21258980/berkeley-coronavirus-covid-19-jesse-arreguin-street-closures

↑, ←, Pg Up, k	Go to previous slide
↓, →, Pg Dn, Space, j	Go to next slide
Home	Go to first slide
End	Go to last slide
Number + Return	Go to specific slide
b / m / f	Toggle blackout / mirrored / fullscreen mode
c	Clone slideshow
p	Toggle presenter mode
t	Restart the presentation timer
?, h	Toggle this help
o	Tile View: Overview of Slides

In-personsession 5

Plan for today

Plan for today

Plan for today

Plan for today

Plan for today

DAGs, continued

Potential outcomesvs. do() notation

Expectations

Outcomes and programs

Causal effects with potential outcomes

Causal effects with do()

do-calculus,adjustment, and CATEs

DAGs and identification

DAGs and identification

DAGs and identification

Easist identification

Most other identification

Unconfoundedness assumption

Logic models, DAGs,and measurement

DAGs vs. Logic models

DAGs vs. Logic models

Plan for today

Help

Class 5: DAGs, do()ing stuff, and measuring stuff

In-personsession 5

Plan for today

Plan for today

Plan for today

Plan for today

Plan for today

DAGs, continued

Potential outcomesvs. do() notation

Expectations

Outcomes and programs

Causal effects with potential outcomes

Causal effects with do()

do-calculus,adjustment, and CATEs

DAGs and identification

DAGs and identification

DAGs and identification

Easist identification

Most other identification

Unconfoundedness assumption

Logic models, DAGs,and measurement

DAGs vs. Logic models

DAGs vs. Logic models

In-person
session 5

Potential outcomes
vs. do() notation

do-calculus,
adjustment, and CATEs

Logic models, DAGs,
and measurement

In-person
session 5

Potential outcomes
vs. do() notation

do-calculus,
adjustment, and CATEs

Logic models, DAGs,
and measurement