class: center middle main-title section-title-3 # In-person<br>session 4 .class-info[ **September 12, 2022** .light[PMAP 8521: Program evaluation<br> Andrew Young School of Policy Studies ] ] --- name: outline class: title title-inv-8 # Plan for today -- .box-4.medium.sp-after-half[Quick useful R tips] -- .box-1.medium.sp-after-half[Logic models] -- .box-2.medium.sp-after-half[Measuring outcomes] -- .box-5.medium[DAGs] --- layout: false name: final-project class: center middle section-title section-title-6 animated fadeIn # Final project details --- layout: false name: r-stuff class: center middle section-title section-title-4 animated fadeIn # Quick useful<br>R tips --- layout: true class: middle --- .box-4.large[Visual markdown editor] --- .box-4.large[Weird figure/table<br>placement in PDFs] ??? https://bookdown.org/yihui/rmarkdown-cookbook/figure-placement.html --- .box-4.large[Figure and table<br>captions and numbers] ??? - <https://bookdown.org/yihui/rmarkdown-cookbook/figure-number.html> - <https://rmd4sci.njtierney.com/figures-tables-captions-.html> - <https://rmd4sci.njtierney.com/start.html> --- class: middle .box-4.large[Side-by-side regression tables] --- <table style="font-size:0.5em;border-bottom: 0; width: auto !important; margin-left: auto; margin-right: auto;" class="table"> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:center;"> Model 1 </th> <th style="text-align:center;"> Model 2 </th> <th style="text-align:center;"> Model 3 </th> <th style="text-align:center;"> Model 4 </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> (Intercept) </td> <td style="text-align:center;"> 362.307 </td> <td style="text-align:center;"> −5780.831*** </td> <td style="text-align:center;"> −5736.897*** </td> <td style="text-align:center;"> −5433.534*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (283.345) </td> <td style="text-align:center;"> (305.815) </td> <td style="text-align:center;"> (307.959) </td> <td style="text-align:center;"> (286.558) </td> </tr> <tr> <td style="text-align:left;"> bill_length_mm </td> <td style="text-align:center;"> 87.415*** </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 6.047 </td> <td style="text-align:center;"> −5.201 </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (6.402) </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (5.180) </td> <td style="text-align:center;"> (4.860) </td> </tr> <tr> <td style="text-align:left;"> flipper_length_mm </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 49.686*** </td> <td style="text-align:center;"> 48.145*** </td> <td style="text-align:center;"> 48.209*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (1.518) </td> <td style="text-align:center;"> (2.011) </td> <td style="text-align:center;"> (1.841) </td> </tr> <tr> <td style="text-align:left;"> sexmale </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 358.631*** </td> </tr> <tr> <td style="text-align:left;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> (41.572) </td> </tr> <tr> <td style="text-align:left;"> Num.Obs. </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 333 </td> </tr> <tr> <td style="text-align:left;"> R2 </td> <td style="text-align:center;"> 0.354 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.760 </td> <td style="text-align:center;"> 0.807 </td> </tr> <tr> <td style="text-align:left;"> R2 Adj. </td> <td style="text-align:center;"> 0.352 </td> <td style="text-align:center;"> 0.758 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.805 </td> </tr> <tr> <td style="text-align:left;"> AIC </td> <td style="text-align:center;"> 5400.0 </td> <td style="text-align:center;"> 5062.9 </td> <td style="text-align:center;"> 5063.5 </td> <td style="text-align:center;"> 4863.3 </td> </tr> <tr> <td style="text-align:left;"> BIC </td> <td style="text-align:center;"> 5411.5 </td> <td style="text-align:center;"> 5074.4 </td> <td style="text-align:center;"> 5078.8 </td> <td style="text-align:center;"> 4882.4 </td> </tr> <tr> <td style="text-align:left;"> Log.Lik. </td> <td style="text-align:center;"> −2696.987 </td> <td style="text-align:center;"> −2528.427 </td> <td style="text-align:center;"> −2527.741 </td> <td style="text-align:center;"> −2426.664 </td> </tr> <tr> <td style="text-align:left;"> F </td> <td style="text-align:center;"> 186.443 </td> <td style="text-align:center;"> 1070.745 </td> <td style="text-align:center;"> 536.626 </td> <td style="text-align:center;"> 457.118 </td> </tr> <tr> <td style="text-align:left;"> RMSE </td> <td style="text-align:center;"> 643.54 </td> <td style="text-align:center;"> 393.12 </td> <td style="text-align:center;"> 392.34 </td> <td style="text-align:center;"> 353.66 </td> </tr> </tbody> <tfoot><tr><td style="padding: 0; " colspan="100%"> <sup></sup> + p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001</td></tr></tfoot> </table> --- <table style="NAborder-bottom: 0; width: auto !important; margin-left: auto; margin-right: auto;" class="table"> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:center;"> Model 1 </th> <th style="text-align:center;"> Model 2 </th> <th style="text-align:center;"> Model 3 </th> <th style="text-align:center;"> Model 4 </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> (Intercept) </td> <td style="text-align:center;"> 362.307 </td> <td style="text-align:center;"> −5780.831*** </td> <td style="text-align:center;"> −5736.897*** </td> <td style="text-align:center;"> −5433.534*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (283.345) </td> <td style="text-align:center;"> (305.815) </td> <td style="text-align:center;"> (307.959) </td> <td style="text-align:center;"> (286.558) </td> </tr> <tr> <td style="text-align:left;"> bill_length_mm </td> <td style="text-align:center;"> 87.415*** </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 6.047 </td> <td style="text-align:center;"> −5.201 </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (6.402) </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (5.180) </td> <td style="text-align:center;"> (4.860) </td> </tr> <tr> <td style="text-align:left;"> flipper_length_mm </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 49.686*** </td> <td style="text-align:center;"> 48.145*** </td> <td style="text-align:center;"> 48.209*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (1.518) </td> <td style="text-align:center;"> (2.011) </td> <td style="text-align:center;"> (1.841) </td> </tr> <tr> <td style="text-align:left;"> sexmale </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 358.631*** </td> </tr> <tr> <td style="text-align:left;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> (41.572) </td> </tr> <tr> <td style="text-align:left;"> Num.Obs. </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 333 </td> </tr> <tr> <td style="text-align:left;"> R2 </td> <td style="text-align:center;"> 0.354 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.760 </td> <td style="text-align:center;"> 0.807 </td> </tr> <tr> <td style="text-align:left;"> R2 Adj. </td> <td style="text-align:center;"> 0.352 </td> <td style="text-align:center;"> 0.758 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.805 </td> </tr> <tr> <td style="text-align:left;"> AIC </td> <td style="text-align:center;"> 5400.0 </td> <td style="text-align:center;"> 5062.9 </td> <td style="text-align:center;"> 5063.5 </td> <td style="text-align:center;"> 4863.3 </td> </tr> <tr> <td style="text-align:left;"> BIC </td> <td style="text-align:center;"> 5411.5 </td> <td style="text-align:center;"> 5074.4 </td> <td style="text-align:center;"> 5078.8 </td> <td style="text-align:center;"> 4882.4 </td> </tr> <tr> <td style="text-align:left;"> Log.Lik. </td> <td style="text-align:center;"> −2696.987 </td> <td style="text-align:center;"> −2528.427 </td> <td style="text-align:center;"> −2527.741 </td> <td style="text-align:center;"> −2426.664 </td> </tr> <tr> <td style="text-align:left;"> F </td> <td style="text-align:center;"> 186.443 </td> <td style="text-align:center;"> 1070.745 </td> <td style="text-align:center;"> 536.626 </td> <td style="text-align:center;"> 457.118 </td> </tr> <tr> <td style="text-align:left;"> RMSE </td> <td style="text-align:center;"> 643.54 </td> <td style="text-align:center;"> 393.12 </td> <td style="text-align:center;"> 392.34 </td> <td style="text-align:center;"> 353.66 </td> </tr> </tbody> <tfoot><tr><td style="padding: 0; " colspan="100%"> <sup></sup> + p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001</td></tr></tfoot> </table> --- class: middle .box-4.large[See full documentation and<br>examples for `modelsummary()` [here](https://vincentarelbundock.github.io/modelsummary/articles/modelsummary.html)] --- layout: true class: middle --- .box-4.large[Make nicer tables when<br>knitting with `kable()`] .box-inv-4[(Or even fancier tables with [kableExtra](https://cran.r-project.org/web/packages/kableExtra/vignettes/awesome_table_in_html.html)!)] --- layout: false name: logic-models class: center middle section-title section-title-1 animated fadeIn # Logic models --- layout: true class: middle --- .box-1.large[Logic models as managerial tools] --- .box-1.large[Inputs vs. Activities vs.<br>Outputs vs. Outcomes] --- .box-1.large[Impact theory vs. logic model] --- layout: false class: title title-1 # Impact theory .box-inv-1[Ensure that the theory linking activities to the outcomes is sound!] .center[ <figure> <img src="img/03/psd-impact-theory-arrow.png" alt="PSD impact theory" title="PSD impact theory" width="100%"> </figure> ] --- .center[ <figure> <img src="img/01/psd-logic-model.png" alt="PSD logic model" title="PSD logic model" width="70%"> </figure> ] --- layout: false class: title title-1 # MPA/MPP at GSU .pull-left[ <figure> <img src="img/03/mpp-mission@4x.png" alt="MPP mission" title="MPP mission" width="130%"> </figure> ] .pull-right[ <figure> <img src="img/03/mpa-mission@4x.png" alt="MPA mission" title="MPA mission" width="100%"> </figure> ] ??? Inputs: - Students - Curriculum - Money (grants, tuition) - AYSPS itself - Faculty, staff - Technology (iCollege, PAWS) - Infrastructure - Transportation - State regulations - National accreditation standards - Faculty research Activities: - Classes - Group work - Exams - Internships - Job fairs - Commencement - Studying - Tutoring - Office hours - Advising - Attendance Outputs: - Assignments and projects - Grades - Degree - Jobs - Awards - Network - Debt Outcomes: - Leadership - AYS national ranking - Public service motivation - Critical thinking - Efficient workforce - Income - Better society --- layout: true class: middle --- .box-1.medium[Isn't it best to always<br>have an articulated theory?] .box-1.medium[Should implicit theory and articulated theory<br>be the same thing in most cases?] --- .box-1.medium[How much does this evaluation stuff cost?] .box-1.medium[Can you do scaled-down versions<br>of these evaluations?] --- .box-1.medium[What if a program exists already<br>and doesn't have a logic model?] --- .box-1.medium[Why would a program aim for final outcomes<br>that can't be measured?] --- .box-1.medium[What should you do if you find that your<br>theory of change (or logic model in general)<br>is wrong in the middle of the program?<br>Is it ethical to stop or readjust?] --- layout: false name: outcomes class: center middle section-title section-title-2 animated fadeIn # Measuring<br>outcomes --- layout: true class: title title-2 --- # Outcomes and programs .box-2.medium[Outcome variable] .box-inv-2.sp-after-half.SMALL[Thing you're measuring] -- .box-2.medium[Outcome change] .box-inv-2.sp-after-half.SMALL[∆ in thing you're measuring over time] -- .box-2.medium[Program effect] .box-inv-2.SMALL[∆ in thing you're measuring over time *because of* the program] --- # Outcomes and programs <figure> <img src="img/04/program-effect.png" alt="Outcomes and program effect" title="Outcomes and program effect" width="100%"> </figure> --- layout: false class: middle .box-2.large[Abstraction] --- layout: false name: dags class: center middle section-title section-title-5 animated fadeIn # DAGs --- layout: true class: middle --- .box-5.large[Causal thinking is necessary—<br>even for descriptive work!] --- .center[ <figure> <img src="img/04-class/supercentenarians.png" alt="Supercentenarians" title="Supercentenarians" width="46%"> </figure> ] ??? Necessity of causal thinking: Mention the McElreath tweet on birth certificate introduction and death ages: <https://twitter.com/rlmcelreath/status/1427564280744976384> <https://www.biorxiv.org/content/10.1101/704080v2> --- layout: false .box-5.less-medium["Every time I get a haircut, I become more mature!"] <figure> <img src="img/04-class/ben-hair.jpg" alt="Benjamin haircut" title="Benjamin haircut" width="100%"> </figure> --- .box-5.less-medium["Every time I get a haircut, I become more mature!"] <img src="04-class_files/figure-html/unnamed-chunk-3-1.png" width="70%" style="display: block; margin: auto;" /> .center[$$E[\text{Maturity} \mid \operatorname{do} (\text{Get haircut})]$$] --- .box-5.less-medium[Getting older opens a backdoor path] <img src="04-class_files/figure-html/unnamed-chunk-4-1.png" width="70%" style="display: block; margin: auto;" /> --- class: middle .box-5.medium[But what does that mean,<br>"opening a backdoor path"?] .box-5.medium[How does statistical association<br>get passed through paths?] --- .box-5.less-medium.sp-after[How do I know which of these is which?] .center[ <figure> <img src="img/05-class/dag-associations.png" alt="DAG associations" title="DAG associations" width="100%"> </figure> ] --- class: middle .pull-left[ <figure> <img src="img/04-class/slider-switch-plain-80.jpg" alt="Switch and slider" title="Switch and slider" width="100%"> </figure> ] .pull-right[ <img src="04-class_files/figure-html/confounding-dag-alone-1.png" width="100%" style="display: block; margin: auto;" /> ] --- class: middle .pull-left[ <figure> <img src="img/04-class/slider-switch-plain-80.jpg" alt="Switch and slider" title="Switch and slider" width="100%"> </figure> ] .pull-right[ <img src="04-class_files/figure-html/mediating-dag-alone-1.png" width="100%" style="display: block; margin: auto;" /> ] --- class: middle .pull-left[ <figure> <img src="img/04-class/slider-switch-plain-80.jpg" alt="Switch and slider" title="Switch and slider" width="100%"> </figure> ] .pull-right[ <img src="04-class_files/figure-html/colliding-dag-alone-1.png" width="100%" style="display: block; margin: auto;" /> ] --- layout: true class: middle --- .box-5.medium[d-separation] .box-inv-5[Except for the one arrow between X and Y,<br>no statistical association can flow between X and Y] .box-inv-5[This is **identification**—<br>all alternative stories are ruled out<br>and the relationship is isolated] --- .box-5.large[How exactly do colliders<br>mess up your results?] .box-5.medium[It looks like you can<br>still get the effect of X on Y] --- <img src="04-class_files/figure-html/nba-dag-1.png" width="65%" style="display: block; margin: auto;" /> --- .center[ <figure> <img src="img/04-class/facebook.png" alt="Facebook collider" title="Facebook collider" width="55%"> </figure> ] ??? <https://www.businessinsider.com/facebook-sent-incomplete-misinformation-data-flawed-researchers-2021-9> <https://www.nytimes.com/live/2020/2020-election-misinformation-distortions#facebook-sent-flawed-data-to-misinformation-researchers> --- layout: false class: title title-5 # Does niceness improve appearance? .pull-left[ <img src="04-class_files/figure-html/unnamed-chunk-5-1.png" width="100%" style="display: block; margin: auto;" /> ] -- .pull-right[ <img src="04-class_files/figure-html/unnamed-chunk-6-1.png" width="100%" style="display: block; margin: auto;" /> ] --- class: title title-5 # Collider distorts the true effect! .pull-left[ <img src="04-class_files/figure-html/unnamed-chunk-7-1.png" width="100%" style="display: block; margin: auto;" /> ] -- .pull-right[ <img src="04-class_files/figure-html/unnamed-chunk-8-1.png" width="100%" style="display: block; margin: auto;" /> ] --- .box-5.large[Effect of race on police use of force<br>using administrative data] --- .box-5.medium[Effect of race on police use of force<br>using administrative data] .pull-left[ <figure> <img src="img/05-class/klm-dag.png" alt="Use of force" title="Use of force" width="100%"> </figure> ] .pull-right[ <figure> <img src="img/05-class/klm.png" alt="Use of force" title="Use of force" width="100%"> </figure> ] --- layout: true class: middle --- .box-5.large[Smoking → Cardiac arrest example] ??? | Person | Smoker | Cardiac arrest | Cholesterol | Weight | Lifestyle healthiness | |--------|--------|----------------|-------------|--------|-----------------------| | 1 | TRUE | TRUE | 150 | 170 | 6 | | 2 | TRUE | FALSE | 170 | 180 | 3 | | 3 | FALSE | FALSE | 130 | 110 | 9 | | 4 | FALSE | TRUE | 140 | 140 | 8 | | 5 | TRUE | TRUE | 120 | 150 | 2 | | 6 | TRUE | FALSE | 130 | 230 | 3 | | 7 | FALSE | FALSE | 140 | 250 | 10 | ```text dag { bb="0,0,1,1" "Cardiac arrest" [outcome,pos="0.599,0.432"] Cholesterol [pos="0.415,0.440"] Lifestyle [pos="0.156,0.317"] Smoking [exposure,pos="0.243,0.428"] Weight [adjusted,pos="0.297,0.255"] Cholesterol -> "Cardiac arrest" Lifestyle -> Smoking Lifestyle -> Weight Smoking -> Cholesterol Weight -> Cholesterol } ``` --- .box-5.medium[How can you be sure<br>you include everything in a DAG?] .box-5.medium[How do you know when to stop?] .box-5.medium[Is there a rule of thumb<br>for the number of nodes?] --- .box-5.medium[Why can we combine nodes in a DAG if they<br>don't represent the same concept?] .box-5.medium[Why include unmeasurable things in a DAG?] --- .box-5.medium[Why do DAGs have to be acyclic?] .box-5.medium[What if there really is reverse causation?] --- .box-5.large[How do we actually<br>adjust for these things?] --- .box-5.large[What's the difference between<br>logic models and DAGs?] .box-inv-5[Can't I just remake my logic model in Dagitty and be done?] --- layout: false class: title title-5 # DAGs vs. Logic models .box-5.large[DAGs are a *statistical* tool] .box-inv-5[Describe a data-generating process<br>and isolate/identify relationships] -- .box-5.large.sp-before[Logic models are a *managerial* tool] .box-inv-5[Oversee the inner workings of a program and its theory] --- class: middle .pull-left[ <figure> <img src="img/04-class/greenspace-eater.png" alt="Green space in Berkeley" title="Green space in Berkeley" width="100%"> </figure> ] .pull-right[ <figure> <img src="img/04-class/greenspace-conversation.png" alt="Covid green spaces" title="Covid green spaces" width="100%"> </figure> ] ??? <https://theconversation.com/how-cities-can-add-accessible-green-space-in-a-post-coronavirus-world-139194> <https://sf.eater.com/2020/5/14/21258980/berkeley-coronavirus-covid-19-jesse-arreguin-street-closures>