Research Methods for Program Evaluation

Author
Affiliation

Kadir Jun Ayhan

Department of Political Science

Published

August 19, 2025

Syllabus (2025 Fall)

Course Title Research Methods for Program Evaluation (PUAD 605)
Department Department of Political Science
Class time & Classroom Monday 6:00 PM – 8:30 PM / Miller 2140 and Miller 2101 (Lab - only for Weeks 5 and 6)
Instructor Dr. Kadir Jun Ayhan
E-mail : ayhankx jmu.edu Phone: : 540-568-5428
Office Hours & Location By appointment / Virtual

1 Course Overview

1.1 Course Description

This course introduces students to the core principles and research methods used in program evaluation. The main learning objective of this course is for students to be able to apply understanding of causal inference for program evaluation to real-world settings, and analyze and critique these cases. I designed this course primarily for graduate students in public administration, policy, and related fields. The course equips students with foundational knowledge in program evaluation, causal inference, and the application of basic methodological tools.

Students begin by exploring theories of change and logic models, which are foundational to program evaluation. The course then guides students through the process of formulating research questions, developing testable hypotheses, and conducting literature reviews to situate their work within existing scholarship.

A key component of the course is the development of computational skills through hands-on experience with R, a widely used statistical programming language. Students will learn the basics of R, including data wrangling and data visualization, and apply these skills to both real-world and hypothetical evaluation scenarios. These skills are highly transferable and valued across academic, public, private, and nonprofit sectors.

The course also covers essential topics in research design, such as describing variables, analyzing relationships, and understanding causal inference through identification strategies and causal diagrams.

I will introduce students to key statistical techniques including regressions, as well as qualitative approaches to causal inference including process tracing with a focus on their application in program evaluation contexts.

Working in teams, students will produce a research paper and present their proposals in Week 13 to receive feedback before submitting their final papers.

By the end of the course, students will have developed a comprehensive understanding of how to design and implement data-driven evaluations and a strong intiution for causal inference.

1.2 Prerequisites

I expect students to have at least introduction-level knowledge of research methods to be successful in this course.

I do not expect students to have any prior knowledge of R or any other programming language. The main purpose of this course is to introduce research methods for program evaluation. R is a tool that we will use to that end. While learning R is not the main goal of this course, it will be a major side benefit of this course.

Note

If you are interested in learning R, I recommend the following resources:

Book for Basics of R: R for Data Science

Book for Data Visualization in R: Hands-on Data Visualization

Basics of R Course: Harvard on Edx - Data Science: R Basics

R Course for JMU students: Introduction to R

Hands-on-practice for R: Datacamp

For more free R books, see Dr. Mine Dogucu’s website.

1.3 Learning Objectives

By the end of this course, students will be able to:

  • Understand key concepts in program evaluation, including theories of change, logic models, and causal inference.

  • Understand practices for sampling and data collection.

  • Conduct literature reviews to support the development of research questions and hypotheses.

  • Implement quantitative and qualitative research methods in the context of program evaluation.

  • Examine relationships between variables using statistical techniques.

  • Assess the validity of causal claims using identification strategies and causal diagrams.

  • Interpret statistical outputs and visualizations.

  • Apply understanding of causal inference for program evaluation to real-world settings, and analyze and critique these cases.

  • Design research for a program evaluation project using R and appropriate statistical methods.

  • Present research findings clearly and effectively to diverse audiences.

  • Use R to perform data wrangling, visualization, and exploratory and causal data analysis.

1.4 Assignments and Evaluation

Proposal presentation (Team) Mid-term Report Final Paper (Team)
10% 50% 40%

Letter Grades: A= 96-100; A-= 91-95; B+= 88-90; B= 84-87; B-= 83-80; C+= 75-79; C= 70-74; C-= 65-69; D+= 60-64; D= 55-59; D-= 50-54; F= below 50.

Explanation of the evaluation system:

1.4.1 IRB Certification

Due date: Sep 08 11:59 PM

All students must complete the IRB Certification before Week 3. This certification is essential for understanding ethical considerations in research involving human subjects. I will not grade you on this assignment, but it is still a requirement for this course.

1.4.2 Proposal Presentation (Team)

Length: 7 minutes presentation, 8 minutes Q/A.

[More information to be added]

1.4.3 Mid-term Report

Due date: Nov 03 Class time

[More information to be added]

1.4.4 Final Paper (Team)

Due date: Dec 08 11:59 PM

[More information to be added]

2 Course Materials and Additional Readings

2.1 Textbooks

The Effect: An Introduction to Research Design and Causality, 2nd Edition (Available Online)

Impact Evaluation in Practice, 2nd Edition

R for Data Science: Import, Tidy, Transform, Visualize, and Model Data (Available Online)

2.2 Supplementary Materials

  • Llaudet, E., & Imai, K. (2022). Data analysis for social science: A friendly and practical introduction. Princeton University Press.

  • Belcher, W. L. (2019). Writing your journal article in twelve weeks: A guide to academic publishing success (2nd ed.). University of Chicago Press.

  • Graff, G., & Birkenstein, C. (2018). They Say / I Say: The moves that matter in academic writing (4th ed.). W.W. Norton & Company.

3 Course Policies

3.1 Contacting Me

Please contact me via email if you have any questions or would like to make an appointment.

3.2 Attendance Policy

Attendance is mandatory. I expect students to attend all classes and participate actively in discussions. If you miss a class, it is your responsibility to catch up on the material covered.

3.3 Academic Honesty

Students must use proper citation and avoid plagiarism. Please use the APA citation style. Plagiarism will not be tolerated and severely punished.

Plagiarism: presenting others’ work without adequate acknowledgement of its source, as though it were one’s own. Plagiarism is a form of fraud. We all stand on the shoulders of others, and we must give credit to the creators of the works that we incorporate into products that we call our own. Some examples of plagiarism:

a sequence of words incorporated without quotation marks;

an unacknowledged passage paraphrased from another’s word;

the use of ideas, sound recordings, computer data or images created by others as though it were one’s own.”

Source.

See also this link.

3.3.1 Generative AI Tools1

I permit you to use generative AI tools for “planning, organizing, outlining, brainstorming, and copy-editing your work for any major or minor assignment”, but this use must be responsible, ethical, and aligned with the goals of academic learning. AI tools should support your understanding, not replace your critical thinking or original contributions. Even when using AI, your assignments must reflect your own voice, reasoning, and ownership.

Think of AI tools as similar to using a dictionary or the internet: helpful for learning and refinement, but not a substitute for your intellectual work.

Please do not prompt AI tools to write your assignments for you. These tools, while powerful, do not understand the context of your assignment or the specific expectations of this course. They generate text based on patterns in training data, not on critical or creative thinking. As such, they often produce generic, superficial, or biased content, and may even hallucinate facts or logic.

3.3.1.1 Al Statement

If you choose to incorporate an Al tool into your writing process, you are required to include a separate “AI Statement” within your assignment. Think of this statement as a brief reflection essay. It should be about [200-250] words in length. In it, you should not only describe your experience of using the AI tool but, more importantly, convincingly convey the significant insights and lessons you gained through the process. These insights should be on par with what you might have learned had you completed the assignment without Al assistance. The assignment will be graded in light of your explanations in the Al statement, thus it is important to persuade the reader that your use of the Al tools provided a valuable learning experience. (Source: Prompted by David McGraw, a JMU ISAT faculty member, ChatGPT developed this syllabus statement for use in his courses.)

3.3.1.2 Proofreading Using AI Tools

You may use AI tools to proofread your assignments. However, the final submission must reflect your own understanding and voice.

To maintain control over your work:

  • Ask AI tools to review small sections (e.g., a paragraph or sentence) rather than entire documents.

  • Carefully review and evaluate any suggestions.

  • If you are unsure about a recommendation, trust your own judgment.

Remember: your cognitive abilities and contextual understanding are superior to those of AI tools.

3.3.1.3 Coding Support from AI Tools

This course introduces basic R programming, but we will not have time to cover all topics in depth.

When you need help with R:

  • Start with the R for Data Science book.

  • Search online (e.g., Stack Overflow, R-bloggers).

  • If you are still stuck, you may consult an AI tool.

When using AI for coding:

  • Avoid asking for full code chunks.

  • Instead, engage the AI in a Socratic dialogue.

    • Begin your prompt by explaining the methodological context of your task, then ask for help step-by-step.

    • Only use code you fully understand. If you cannot explain any line of code, you should not include it in your assignment.

Note

Foundational Prompt (add this paragraph to every question):

II would like to translate my understanding of research methods for program evaluation into coding in R. As a beginner in R, I need your help specifically with how to implement the statistical techniques I want to apply, rather than simply providing full code chunks.

Please guide me step by step, asking whether I understand what I want to do next with each line of code. Use a tone that is friendly and supportive of a novice learner, while also offering enough depth to help me gradually build coding skills based on the research methods I already understand.

Additionally, please incorporate principles of ethical research and encourage active cognitive processing throughout your guidance.

Example Prompt (following the previous paragraph):

With this in mind, could you help me perform a t-test on my dataset using R?

1 “Generative AI (GenAI) is a form of artificial intelligence capable of producing new content using predictive algorithms. Text-based GenAI tools like OpenAI’s ChatGPT or Google’s Gemini are powered by Large Language Models (LLMs). LLMs are machine learning models pre-trained with large amounts of data to learn patterns and norms. In response to a user’s prompt, GenAI uses those learned patterns to predict and create plausible outputs. These are predictive models that generate new content based on learned patterns so no output will be the same (even though sometimes outputs have a common style or tone).” Source

3.3.2 JMU Honor Code

The JMU Honor Code is available from the Honor Council Web site.

3.3.3 JMU SafeAssign

In this course one or more of your writing assignments may be submitted to the instructor through Blackboard’s SafeAssign plagiarism prevention service as approved by JMU. Your writing assignment will be checked for plagiarism against Internet sources, millions of academic journal articles, the JMU SafeAssign database and the SafeAssign Global Reference Database. SafeAssign generates an originality report for the instructor that highlights any blocks of text in your paper that match the above reference sources and allows a line-by-line comparison of potentially unoriginal text from your paper with the matching document sections in the reference sources. Each paper you submit through SafeAssign for this or any class at JMU will be added to the JMU SafeAssign database and later used only to check against other JMU paper submissions. Neither Blackboard nor JMU claim any copyright ownership of your writing submitted through SafeAssign. When you submit your paper through SafeAssign you will be given the choice of whether or not to “opt in” and permanently contribute a copy of your paper to Blackboard’s Global Reference Database. This would protect your original writing from plagiarism at other institutions. Opting in and voluntarily contributing your work to the global database is an individual student decision and not required by your instructor or JMU. For more information about SafeAssign refer to the Web site.

3.4 Late Submissions

For late submissions, you will get 80% of your grading unless you have a valid excuse. You can always contact me if you have a valid excuse to ask for an extension. I do not require students to submit official documents (doctor report etc.). Your word is enough for me.

3.5 Adding/ Dropping Classes

Students are responsible for registering for classes and for verifying their class schedules on e-campus. Please refer to the JMU Academic Calendar for the last day to add or drop classes.

3.6 Disability Accommodations

If you need an accommodation based on the impact of a disability, you should contact the Office of Disability Services (Wilson Hall, Room 107, 540-568-6705) if you have not previously done so. Disability Services will provide you with an Access Plan Letter that will verify your need for services and make recommendations for accommodations to be used in the classroom. Once you have presented me with this letter, you and I will sit down and review the course requirements, your disability characteristics, and your requested accommodations to develop an individualized plan, appropriate for PUAD 605.

3.7 Inclement Weather Policies

JMU’s cancellation policy.

3.8 Religious Observation Accommodations

All faculty are required to give reasonable and appropriate accommodations to students requesting them on grounds of religious observation. The faculty member determines what accommodations are appropriate for his/her course. Students should notify the faculty by no later than the end of the Drop-Add period the first week of the semester of potential scheduled absences and deter¬mine with the instructor if mutually acceptable alternative methods exist for completing the missed classroom time, lab or activity.

3.9 Special Needs

If you have other special needs, please let me know. I will do my best to flexibly accommodate your needs.

4 Student Resources

Please refer to the JMU Student Resources for additional resources available to students, including.

5 Course Schedule

Week Date Topics & Class Materials, Assignments
1 Aug 25 Introduction of the Course
2 Sep 01 Designing Research and Asking Research Questions
3 Sep 08 Literature Review
4 Sep 15 Theories of Change and Logic Model
5 Sep 22 Basics of R, Data Wrangling
6 Sep 29 Data Visualization
7 Oct 06 Variables and Relationships
8 Oct 13 Causal Diagrams 1
9 Oct 20 Causal Diagrams 2
10 Oct 27 Process Tracing
11 Nov 03 Exploratory Data Analysis
12 Nov 10 Randomized Experiments
13 Nov 17 Research Proposal Team Presentations and Feedback
14 Nov 24 Thanksgiving Break
15 Dec 01 Review and Debriefing
16 Dec 08 Submission of Final Papers

6 Detailed Course Schedule with Readings

Week 1 (Aug 25): Introduction of the Course

Introduction of the course including course contents, assignments, and expectations.

Week 2 (Sep 01): Designing Research and Asking Research Questions

Required Readings:

  • Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman; Hall/CRC. https://theeffectbook.net/ Chapters 1, 2

  • Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact evaluation in practice (2nd ed.). Inter-American Development Bank; World Bank. https://doi.org/10.1596/978-1-4648-0779-4 Chapter 1

Recommended Readings:

  • Belcher, W. L. (2019). Writing your journal article in twelve weeks: A guide to academic publishing success (2nd ed.). University of Chicago Press. Week 2 Chapter

Week 3 (Sep 08): Literature Review

Required Readings:

  • Belcher, W. L. (2019). Writing your journal article in twelve weeks: A guide to academic publishing success (2nd ed.). University of Chicago Press. Week 5 Chapter

  • Marzi, G., Balzano, M., Caputo, A., & Pellegrini, M. M. (2025). Guidelines for bibliometric-systematic literature reviews: 10 steps to combine analysis, synthesis and theory development. International Journal of Management Reviews, 27(1), 81–103. https://doi.org/https://doi.org/10.1111/ijmr.12381

Recommended Readings:

  • Graff, G., & Birkenstein, C. (2018). They Say / I Say: The moves that matter in academic writing (4th ed.). W.W. Norton & Company. Parts 1 and 2.

Some Literature Review Tools:

Week 4 (Sep 15): Theories of Change and Logic Model

Required Readings:

  • Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact evaluation in practice (2nd ed.). Inter-American Development Bank; World Bank. https://doi.org/10.1596/978-1-4648-0779-4 Chapter 2

  • Cuhadar, E., Genc, O. G., & Kotelis, A. (2015). A Greek–Turkish peace project: Assessing the effectiveness of interactive conflict resolution. Southeast European and Black Sea Studies, 15(4), 563–583. https://doi.org/10.1080/14683857.2015.1020141

Recommended Readings:

Week 5 (Sep 22): Basics of R, Data Wrangling

Readings:

  • Ayhan, K. J. (2024). R for Korean Studies: A gentle introduction to Computational Social Science (Draft Version 0.0.2). https://r4ks.com Chapter 2

  • Wickham, H., & Grolemund, G. (2023). R for data science (2nd ed.). O’Reilly Media. https://r4ds.hadley.nz/ Chapters 2, 3, 5, 6, 7, and 8

Week 6 (Sep 29): Data Visualization

Readings:

  • Wickham, H., & Grolemund, G. (2023). R for data science (2nd ed.). O’Reilly Media. https://r4ds.hadley.nz/ Chapters 1, 9, 10, and 11

Week 7 (Oct 06): Variables and Relationships

Readings:

  • Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman; Hall/CRC. https://theeffectbook.net/ Chapters 3 and 4

Week 8 (Oct 13): Causal Diagrams 1

Required Readings:

  • Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman; Hall/CRC. https://theeffectbook.net/ Chapters 5, 6 and 7

Recommended Readings:

  • Rohrer, J. M. (2018). Thinking clearly about correlations and causation: Graphical causal models for observational data. Advances in Methods and Practices in Psychological Science, 1(1), 27–42. https://doi.org/10.1177/2515245917745629

Week 9 (Oct 20): Causal Diagrams 2

Required Readings:

  • Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman; Hall/CRC. https://theeffectbook.net/ Chapters 8, 9 and 10

Week 10 (Oct 27): Process Tracing

Required Readings:

Recommended Readings:

Week 11 (Nov 03): Exploratory Data Analysis

Readings:

  • Llaudet, E., & Imai, K. (2022). Data analysis for social science: A friendly and practical introduction. Princeton University Press. Chapter 3

Week 12 (Nov 10): Randomized Experiments

Readings:

  • Llaudet, E., & Imai, K. (2022). Data analysis for social science: A friendly and practical introduction. Princeton University Press. Chapter 2

  • Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact evaluation in practice (2nd ed.). Inter-American Development Bank; World Bank. https://doi.org/10.1596/978-1-4648-0779-4 Chapter 4

  • Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman; Hall/CRC. https://theeffectbook.net/ Chapter 13

Week 13 (Nov 17): Research Proposal Team Presentations and Feedback

Week 14 (Nov 24): Thanksgiving Break

Week 15 (Dec 01): Review and Debriefing

Week 16 (Dec 08): Submission of Final Papers

Notes

The contents of this syllabus are not final. I may update them later.

Read the Syllabus!