Week | Date | Topics & Class Materials, Assignments |
---|---|---|
1 | Aug 25 | Introduction of the Course: The Syllabus and Learning Objectives |
2 | Sep 01 | Designing Research and Asking Research Questions |
3 | Sep 08 | Research Design Basics: Literature Review |
4 | Sep 15 | Theories of Change and Logic Model |
5 | Sep 22 | Introduction to R for Program Evaluation: Basics of R, Data Wrangling |
6 | Sep 29 | Introduction to R for Program Evaluation: Data Visualization |
7 | Oct 06 | Research Design Basics: Variables and Relationships |
8 | Oct 13 | Introduction to Causal Inference: Causal Diagrams |
9 | Oct 20 | Introduction to Causal Inference: Process Tracing |
10 | Oct 27 | Exploratory Data Analysis: Survey Research |
11 | Nov 03 | Randomized Experiments |
12 | Nov 10 | Difference in differences |
13 | Nov 17 | Research Proposal Team Presentations and Feedback |
14 | Nov 24 | Thanksgiving Break |
15 | Dec 01 | Review and Debriefing |
16 | Dec 08 | Submission of Final Papers |
Syllabus (2025 Fall)
Course Title | Research Methods for Program Evaluation (PUAD 605) |
Department | Department of Political Science |
Class time & Classroom | Monday 6:00 PM – 8:30 PM / Miller 2140 and Miller 2101 (Lab - only for Weeks 5 and 6) |
Instructor | Dr. Kadir Jun Ayhan |
E-mail : ayhankx jmu.edu | Phone: : 540-568-5428 |
Office Hours & Location | By appointment / Virtual |
1 Course Overview
1.1 Course Description
This course introduces students to the core principles and research methods used in program evaluation. The main learning objective of this course is for students to be able to apply understanding of causal inference for program evaluation to real-world settings, and analyze and critique these cases. I designed this course primarily for graduate students in public administration, policy, and related fields. The course equips students with foundational knowledge in program evaluation, causal inference, and the application of basic methodological tools.
Students begin by exploring theories of change and logic models, which are foundational to program evaluation. The course then guides students through the process of formulating research questions, developing testable hypotheses, and conducting literature reviews to situate their work within existing scholarship.
A key component of the course is the development of computational skills through hands-on experience with R
, a widely used statistical programming language. Students will learn the basics of R, including data wrangling and data visualization, and apply these skills to both real-world and hypothetical evaluation scenarios. These skills are highly transferable and valued across academic, public, private, and nonprofit sectors.
The course also covers essential topics in research design, such as describing variables, analyzing relationships, and understanding causal inference through identification strategies and causal diagrams.
I will introduce students to key statistical techniques including regressions, as well as qualitative approaches to causal inference including process tracing with a focus on their application in program evaluation contexts.
Working in teams, students will produce a research paper and present their proposals in Week 13 to receive feedback before submitting their final papers.
By the end of the course, students will have developed a comprehensive understanding of how to design and implement data-driven evaluations and a strong intiution for causal inference.
1.2 Prerequisites
I expect students to have at least introduction-level knowledge of research methods to be successful in this course.
I do not expect students to have any prior knowledge of R or any other programming language. The main purpose of this course is to introduce research methods for program evaluation. R is a tool that we will use to that end. While learning R is not the main goal of this course, it will be a major side benefit of this course.
If you are interested in learning R, I recommend the following resources:
Book for Basics of R: R for Data Science
Book for Data Visualization in R: Hands-on Data Visualization
Basics of R Course: Harvard on Edx - Data Science: R Basics
R Course for JMU students: Introduction to R
Hands-on-practice for R: Datacamp
For more free R books, see Dr. Mine Dogucu’s website.
1.3 Learning Objectives
By the end of this course, students will be able to:
Understand key concepts in program evaluation, including theories of change, logic models, and causal inference.
Understand practices for sampling and data collection.
Conduct literature reviews to support the development of research questions and hypotheses.
Implement quantitative and qualitative research methods in the context of program evaluation.
Examine relationships between variables using statistical techniques.
Assess the validity of causal claims using identification strategies and causal diagrams.
Interpret statistical outputs and visualizations.
Apply understanding of causal inference for program evaluation to real-world settings, and analyze and critique these cases.
Design research for a program evaluation project using R and appropriate statistical methods.
Present research findings clearly and effectively to diverse audiences.
Use R to perform data wrangling, visualization, and exploratory and causal data analysis.
1.4 Assignments and Evaluation
News assignments | Proposal presentation (Team) | Hypothetical Scenario | Final Paper (Team) |
---|---|---|---|
20% | 10% | 30% | 40% |
Letter Grades: A
= 96-100; A-
= 91-95; B+
= 88-90; B
= 84-87; B-
= 83-80; C+
= 75-79; C
= 70-74; C-
= 65-69; D+
= 60-64; D
= 55-59; D-
= 50-54; F
= below 50.
Explanation of the evaluation system:
1.4.1 IRB Certification
Due date: Sep 08 11:59 PM
All students must complete the IRB Certification before Week 4. This certification is essential for understanding ethical considerations in research involving human subjects. I will not grade you on this assignment, but it is still a requirement for this course.
1.4.2 News Assignments
Find a recent news article (published within the last month) that discusses a problem or a proposed solution to a problem. Based on the article, draw the causal diagram implied by the story. Then, write a short critical review (up to two sentences) evaluating the causal implications presented in the article.
You may submit up to four of these assignments on Canvas before class time throughout the semester.
1.4.3 Proposal Presentation (Team)
Length: 7 minutes presentation, 8 minutes Q/A.
[More information to be added]
1.4.4 Hypothetical Scenario
[More information to be added]
1.4.5 Final Paper (Team)
Due date: Dec 08 11:59 PM
[More information to be added]
2 Course Materials and Additional Readings
2.1 Textbooks
The Effect: An Introduction to Research Design and Causality, 2nd Edition (Available Online)
Impact Evaluation in Practice, 2nd Edition
R for Data Science: Import, Tidy, Transform, Visualize, and Model Data (Available Online)
2.2 Supplementary Materials
Llaudet, E., & Imai, K. (2022). Data analysis for social science: A friendly and practical introduction. Princeton University Press.
Belcher, W. L. (2019). Writing your journal article in twelve weeks: A guide to academic publishing success (2nd ed.). University of Chicago Press.
Graff, G., & Birkenstein, C. (2018). They Say / I Say: The moves that matter in academic writing (4th ed.). W.W. Norton & Company.
3 Course Policies
3.1 Contacting Me
Please contact me via email if you have any questions or would like to make an appointment.
3.2 Attendance Policy
Attendance is mandatory. I expect students to attend all classes and participate actively in discussions. If you miss a class, it is your responsibility to catch up on the material covered.
3.3 Academic Honesty
Students must use proper citation and avoid plagiarism. Please use the APA citation style. Plagiarism will not be tolerated and severely punished.
Plagiarism: presenting others’ work without adequate acknowledgement of its source, as though it were one’s own. Plagiarism is a form of fraud. We all stand on the shoulders of others, and we must give credit to the creators of the works that we incorporate into products that we call our own. Some examples of plagiarism:
a sequence of words incorporated without quotation marks;
an unacknowledged passage paraphrased from another’s word;
the use of ideas, sound recordings, computer data or images created by others as though it were one’s own.”
See also this link.
3.3.1 Generative AI Tools1
I permit you to use generative AI tools for “planning, organizing, outlining, brainstorming, and copy-editing your work for any major or minor assignment”, but this use must be responsible, ethical, and aligned with the goals of academic learning. AI tools should support your understanding, not replace your critical thinking or original contributions. Even when using AI, your assignments must reflect your own voice, reasoning, and ownership.
Think of AI tools as similar to using a dictionary or the internet: helpful for learning and refinement, but not a substitute for your intellectual work.
Please do not prompt AI tools to write your assignments for you. These tools, while powerful, do not understand the context of your assignment or the specific expectations of this course. They generate text based on patterns in training data, not on critical or creative thinking. As such, they often produce generic, superficial, or biased content, and may even hallucinate facts or logic.
3.3.1.1 Al Statement
If you choose to incorporate an Al tool into your writing process, you are required to include a separate “AI Statement” within your assignment. Think of this statement as a brief reflection essay. It should be about [200-250] words in length. In it, you should not only describe your experience of using the AI tool but, more importantly, convincingly convey the significant insights and lessons you gained through the process. These insights should be on par with what you might have learned had you completed the assignment without Al assistance. The assignment will be graded in light of your explanations in the Al statement, thus it is important to persuade the reader that your use of the Al tools provided a valuable learning experience. (Source: Prompted by David McGraw, a JMU ISAT faculty member, ChatGPT developed this syllabus statement for use in his courses.)
3.3.1.2 Proofreading Using AI Tools
You may use AI tools to proofread your assignments. However, the final submission must reflect your own understanding and voice.
To maintain control over your work:
Ask AI tools to review small sections (e.g., a paragraph or sentence) rather than entire documents.
Carefully review and evaluate any suggestions.
If you are unsure about a recommendation, trust your own judgment.
Remember: your cognitive abilities and contextual understanding are superior to those of AI tools.
3.3.1.3 Coding Support from AI Tools
This course introduces basic R programming, but we will not have time to cover all topics in depth.
When you need help with R:
Start with the R for Data Science book.
Search online (e.g., Stack Overflow, R-bloggers).
If you are still stuck, you may consult an AI tool.
When using AI for coding:
Avoid asking for full code chunks.
-
Instead, engage the AI in a Socratic dialogue.
Begin your prompt by explaining the methodological context of your task, then ask for help step-by-step.
Only use code you fully understand. If you cannot explain any line of code, you should not include it in your assignment.
Foundational Prompt (add this paragraph to every question):
II would like to translate my understanding of research methods for program evaluation into coding in R. As a beginner in R, I need your help specifically with how to implement the statistical techniques I want to apply, rather than simply providing full code chunks.
Please guide me step by step, asking whether I understand what I want to do next with each line of code. Use a tone that is friendly and supportive of a novice learner, while also offering enough depth to help me gradually build coding skills based on the research methods I already understand.
Additionally, please incorporate principles of ethical research and encourage active cognitive processing throughout your guidance.
Example Prompt (following the previous paragraph):
With this in mind, could you help me perform a t-test on my dataset using R?
1 “Generative AI (GenAI) is a form of artificial intelligence capable of producing new content using predictive algorithms. Text-based GenAI tools like OpenAI’s ChatGPT or Google’s Gemini are powered by Large Language Models (LLMs). LLMs are machine learning models pre-trained with large amounts of data to learn patterns and norms. In response to a user’s prompt, GenAI uses those learned patterns to predict and create plausible outputs. These are predictive models that generate new content based on learned patterns so no output will be the same (even though sometimes outputs have a common style or tone).” Source
3.3.2 JMU Honor Code
The JMU Honor Code is available from the Honor Council Web site.
3.3.3 JMU SafeAssign
In this course one or more of your writing assignments may be submitted to the instructor through Blackboard’s SafeAssign plagiarism prevention service as approved by JMU. Your writing assignment will be checked for plagiarism against Internet sources, millions of academic journal articles, the JMU SafeAssign database and the SafeAssign Global Reference Database. SafeAssign generates an originality report for the instructor that highlights any blocks of text in your paper that match the above reference sources and allows a line-by-line comparison of potentially unoriginal text from your paper with the matching document sections in the reference sources. Each paper you submit through SafeAssign for this or any class at JMU will be added to the JMU SafeAssign database and later used only to check against other JMU paper submissions. Neither Blackboard nor JMU claim any copyright ownership of your writing submitted through SafeAssign. When you submit your paper through SafeAssign you will be given the choice of whether or not to “opt in” and permanently contribute a copy of your paper to Blackboard’s Global Reference Database. This would protect your original writing from plagiarism at other institutions. Opting in and voluntarily contributing your work to the global database is an individual student decision and not required by your instructor or JMU. For more information about SafeAssign refer to the Web site.
3.4 Late Submissions
For late submissions, you will get 80% of your grading unless you have a valid excuse. You can always contact me if you have a valid excuse to ask for an extension. I do not require students to submit official documents (doctor report etc.). Your word is enough for me.
3.5 Adding/ Dropping Classes
Students are responsible for registering for classes and for verifying their class schedules on e-campus. Please refer to the JMU Academic Calendar for the last day to add or drop classes.
3.6 Disability Accommodations
If you need an accommodation based on the impact of a disability, you should contact the Office of Disability Services (Wilson Hall, Room 107, 540-568-6705) if you have not previously done so. Disability Services will provide you with an Access Plan Letter that will verify your need for services and make recommendations for accommodations to be used in the classroom. Once you have presented me with this letter, you and I will sit down and review the course requirements, your disability characteristics, and your requested accommodations to develop an individualized plan, appropriate for PUAD 605.
3.7 Inclement Weather Policies
3.8 Religious Observation Accommodations
All faculty are required to give reasonable and appropriate accommodations to students requesting them on grounds of religious observation. The faculty member determines what accommodations are appropriate for his/her course. Students should notify the faculty by no later than the end of the Drop-Add period the first week of the semester of potential scheduled absences and deter¬mine with the instructor if mutually acceptable alternative methods exist for completing the missed classroom time, lab or activity.
3.9 Special Needs
If you have other special needs, please let me know. I will do my best to flexibly accommodate your needs.
4 Student Resources
Please refer to the JMU Student Resources for additional resources available to students, including.
5 Course Schedule
6 Detailed Course Schedule with Readings
Week 1 (Aug 25): Introduction of the Course: The Syllabus and Learning Objectives
Introduction of the course including course contents, assignments, and expectations.
Week 2 (Sep 01): Designing Research and Asking Research Questions
Readings:
Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman; Hall/CRC. https://theeffectbook.net/ Chapters 1, 2
Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact evaluation in practice (2nd ed.). Inter-American Development Bank; World Bank. https://doi.org/10.1596/978-1-4648-0779-4 Chapter 1
Belcher, W. L. (2019). Writing your journal article in twelve weeks: A guide to academic publishing success (2nd ed.). University of Chicago Press. Week 2 Chapter
Week 3 (Sep 08): Research Design Basics: Literature Review
Required Readings:
Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact evaluation in practice (2nd ed.). Inter-American Development Bank; World Bank. https://doi.org/10.1596/978-1-4648-0779-4 Chapter 2
Cuhadar, E., Genc, O. G., & Kotelis, A. (2015). A greek–turkish peace project: Assessing the effectiveness of interactive conflict resolution. Southeast European and Black Sea Studies, 15(4), 563–583. https://doi.org/10.1080/14683857.2015.1020141
Recommended Readings:
Shapiro, I. (2005). Theories of change. http://www.beyondintractability.org/essay/theories-of-change
W.K. Kellogg Foundation. (2004). Logic model development guide. W.K. Kellogg Foundation. https://www.betterevaluation.org/sites/default/files/2021-11/Kellogg_Foundation_Logic_Model_Guide.pdf
Week 4 (Sep 15): Theories of Change and Logic Model
Required Readings:
Belcher, W. L. (2019). Writing your journal article in twelve weeks: A guide to academic publishing success (2nd ed.). University of Chicago Press. Week 5 Chapter
Marzi, G., Balzano, M., Caputo, A., & Pellegrini, M. M. (2025). Guidelines for bibliometric-systematic literature reviews: 10 steps to combine analysis, synthesis and theory development. International Journal of Management Reviews, 27(1), 81–103. https://doi.org/https://doi.org/10.1111/ijmr.12381
Recommended Readings:
- Graff, G., & Birkenstein, C. (2018). They Say / I Say: The moves that matter in academic writing (4th ed.). W.W. Norton & Company. Parts 1 and 2.
Some Literature Review Tools:
Week 5 (Sep 22): Introduction to R for Program Evaluation: Basics of R, Data Wrangling
Readings:
Ayhan, K. J. (2024). R for Korean Studies: A gentle introduction to Computational Social Science (Draft Version 0.0.2). https://r4ks.com Chapter 2
Wickham, H., & Grolemund, G. (2023). R for data science (2nd ed.). O’Reilly Media. https://r4ds.hadley.nz/ Chapters 2, 3, 5, 6, 7, and 8
Week 6 (Sep 29): Introduction to R for Program Evaluation: Data Visualization
Readings:
- Wickham, H., & Grolemund, G. (2023). R for data science (2nd ed.). O’Reilly Media. https://r4ds.hadley.nz/ Chapters 1, 9, 10, and 11
Week 7 (Oct 06): Research Design Basics: Variables and Relationships
Readings:
- Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman; Hall/CRC. https://theeffectbook.net/ Chapters 3 and 4
Week 8 (Oct 13): Introduction to Causal Inference: Causal Diagrams
Readings:
Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman; Hall/CRC. https://theeffectbook.net/ Chapters 6 and 7
Rohrer, J. M. (2018). Thinking clearly about correlations and causation: Graphical causal models for observational data. Advances in Methods and Practices in Psychological Science, 1(1), 27–42. https://doi.org/10.1177/2515245917745629
Week 9 (Oct 20): Introduction to Causal Inference: Process Tracing
Required Readings:
Collier, D. (2011). Understanding process tracing. PS: Political Science & Politics, 44(4), 823–830. https://doi.org/10.1017/S1049096511001429
Trampusch, C., & Palier, B. (2016). Between x and y: How process tracing contributes to opening the black box of causality. New Political Economy, 21(5), 437–454. https://doi.org/10.1080/13563467.2015.1134465
Kuru, A. T. (2007). Passive and assertive secularism: Historical conditions, ideological struggles, and state policies toward religion. World Politics, 59(4), 568–594. https://doi.org/10.1353/wp.2008.0005
Recommended Readings:
- Bennett, A., & Checkel, J. T. (Eds.). (2014). Process tracing: From metaphor to analytic tool. Cambridge University Press. https://doi.org/10.1017/CBO9781139858472
Week 10 (Oct 27): Exploratory Data Analysis: Survey Research
Readings:
- Llaudet, E., & Imai, K. (2022). Data analysis for social science: A friendly and practical introduction. Princeton University Press. Chapter 3
Week 11 (Nov 03): Randomized Experiments
Readings:
Llaudet, E., & Imai, K. (2022). Data analysis for social science: A friendly and practical introduction. Princeton University Press. Chapter 2
Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact evaluation in practice (2nd ed.). Inter-American Development Bank; World Bank. https://doi.org/10.1596/978-1-4648-0779-4 Chapter 4
Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman; Hall/CRC. https://theeffectbook.net/ Chapter 13
Week 12 (Nov 10): Difference in differences
Readings:
Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact evaluation in practice (2nd ed.). Inter-American Development Bank; World Bank. https://doi.org/10.1596/978-1-4648-0779-4 Chapter 7
Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman; Hall/CRC. https://theeffectbook.net/ Chapter 18
Week 13 (Nov 17): Research Proposal Team Presentations and Feedback
Week 14 (Nov 24): Thanksgiving Break
Week 15 (Dec 01): Review and Debriefing
Week 16 (Dec 08): Submission of Final Papers
See Final Paper (Team) Section (Section 1.4.5) for details.
Notes
The contents of this syllabus are not final. I may update them later.