Public Affairs 6080: Concepts and Methods of Program Evaluation

This course is designed to address the following John Glenn College learning objectives:

• Assess data quality and create surveys and sampling methods to address problems
• Manage storage and retrieval of data
• Seek and identify patterns in data
• Support claims with statistically sound quantitative and/or qualitative evidence

By the end of the semester, the student should be able to do the following:       

1. Critically assess evaluation plans/research designs and research,
2. Design an outcomes evaluation study, and
3. Think like a policy analyst.

Two quizzes -15% each, 30%

Nine Class Participation and In-Class Exercises (1 point each), 9%

Student Lead Quick Weekly Review (student signups for date), 2%

Seven Discussion Posts (each 1%), 7%                   

• Based on weekly PowerPoints and to be posted prior to class (see Descriptions of Major Course Assignments on pg. 4 for more details).
• No make-ups are available for these assignments.

Evaluation Proposal Project – Preferred as group or partner project but individual projects are permissible with approval, 52%       

• Paper Topic Document (3%)
• Paper Part 1:  Introduction and Process Evaluation Proposal (20%)
• Peer Review Document (2%)
• Paper Part 2:  Outcome Evaluation Proposal (22%)
• Presentation (5%)

Two quizzes will be given worth 15% of your final grade each (total of 30% for both). More information regarding the structure of each quiz and a study guide will be available prior to the quiz. Make-up quizzes will be given only when prior arrangements are made and only for documentable and unavoidable circumstances. Collaboration on quizzes is strictly forbidden.

Based on weekly lectures, I will pose a question and pin the question to the Discussion section of Carmen. Please post a short response to the question prior to the class discussion. These posts do not need to be long (3 sentences to a short paragraph). The goal is to delve into the material and start the discussion prior to class. This also helps me to assess any gaps or questions that you have about the material for class discussion. Make-ups and late submissions are not available for Discussion Posts. If you need to miss a class on a day that a post is due, please be sure to still complete your discussion post prior to class to receive full points.

Discussion Topics

• Program Description
• Process Evaluation
• Conceptual and Methodological Issues
• Experiments
• Quasi experiments I
• Quasi experiments II
• Pre-experiments/Non-experiments

Each of the nine curriculum-based weeks will begin with student lead reviews. Students can sign-up independently or may collaborate (no more than 3 to an assignment) for one weekly review. The review should focus on one or more parts of the lecture/curriculum for that week. Reviews can take any form based on student discretion (e.g., Word Cloud, Kahoot!, Blooket, Gimkit, handout, slides).  Reviews should be quick (aim for no more than 5 minutes).

The following assignments are to be completed with a partner or in groups (individual projects are permitted with approval). You will have time to form groups in Week 1.  I will attempt to allot a few minutes in each class for your group to discuss your project plan, but your group should also communicate outside of class to complete project expectations.

Project Paper Topic Deliverable
One per group - This deliverable should be submitted via upload to Carmen. The deliverable must be in MS Word format (*no* pdf). The intent of the deliverable is to start the discussion regarding your evaluation questions and dataset. This allows me to review your thoughts, provide feedback, and assist with any reworking before you begin to research and compose your papers. Components of this deliverable should include all the following:

• A proposed process evaluation question for your first paper
• A proposed outcome evaluation question for your second paper
• Although your group will not be required to conduct the analysis in your proposal, you will need to identify a dataset to support this project. Your Paper Topic Deliverable should address what *specific* dataset will enable you or someone else to address your outcome evaluation question and ultimately implement the outcome evaluation you will design by the end of this class (include a link or data dictionary of the proposed dataset).
  • ​​​​​​​Where is the dataset available? Is it publicly available?
  • For what years are there data?
  • What variable or variables from this dataset do you think that you will use?
  • What is the level of analysis (e.g., participant-level, program-level, state-level)?
• What are some of the relevant academic journals for the literature review portion of your process and outcome/impact evaluation design?
• Please also attach a pdf of a peer-reviewed research article on your topic.

Project Process Evaluation Design Papers (see guidelines). 
Papers must be uploaded to the appropriate Assignment folder by 11:59 p.m. Please use MS Word, double space, Times New Roman font size 12, 1-inch margins.

Project Peer Review Deliverable
Bring 4 hard copies per group to class. Be clear and succinct in an outline/bullet form.  The document should not exceed one page and must respond to a-d below. Do not provide your entire paper. Your group will share your design with peers via this format, and all will provide comments/thoughts on other groups’ designs. 

• Your outcome evaluation question(s) stated clearly and precisely.
• Indication of your evaluation design, using X and O notation.
• Key variables that you will use to reduce the non-equivalence of the treatment and comparison group(s).
• Strengths/weaknesses of design

Project Student Presentations
Presentations will include at least 5 slides: Intro, Evaluation Question, X O Evaluation design, key variables, strengths/weaknesses of design. Feedback on design provided by instructor and students.

Project Outcome Evaluation Proposal (see guidelines below)
Paper must be uploaded to the appropriate Assignment folder by 11:59 pm. Please use MS Word, double space, Times New Roman font size 12, 1-inch margins.

Students will devise an evaluation proposal (process and outcome) for a program of their choosing. The program may be a local, state, national, or international program/intervention/initiative/expenditure of time, money, and resources for which data are available to assess its outcomes/impact.  It can be a program within a public or nonprofit organization. Again, data availability and previous evaluations of the same/similar program are necessary conditions for the program to be an acceptable focus for this project.  Data from previous evaluations of the same program can serve as supplemental, but not as primary, data for your evaluation design. I am happy to help and can provide a dataset if a group is unable to find one of interest.

Process Evaluation Proposal:
The design should not exceed 12 double-spaced pages (a list of references is not included in the page limit). You must use 12-pt Times New Roman font with 1-inch margins. 

The paper will:

• Include an introduction and description of the program;
• Identify program components (who, what, when, where, and how of the program);
• Clearly state the process evaluation question;
• Include a logic model and explanation of the theory of change underlying that model; 
• Develop a plan (based on the evaluation question) for comparing the program as implemented to the program as represented in its ideal form in the logic model, including a broad discussion of the measures, methods, data collection, timing and proposed resources to complete the evaluation.

Note: A “good” question for your process evaluation could address one or more of the following examples (with more examples provided in class):

1. One which addresses the quality of implementation (e.g., whether there were appropriate strategies, activities and/or resources to support the program’s goals).
2. One which addresses whether program activities were accomplished.
3. One which addresses barriers to implementation.
4. One which addresses whether the target audience was reached.

Outcome Evaluation Proposal:
The design should not exceed 12 double-spaced pages. You must use 12-pt Times New Roman font with 1-inch margins. 

The paper will:

• Clearly state the evaluation question;
• Literature Review (please cite references): Critically assess previous evaluations of the same or similar topic, including but not limited to the design, data used, treatment of the counterfactual, internal and external validity of those evaluations (indicating why threats are more or are less plausible) and the choice of outcome measures used;
• Identify valid, reliable outcome measures and explain their rationale, including consideration of other outcomes, both intended and unintended;
• Propose an evaluation design using notation discussed in class;
• Indicate how you will address the counterfactual and relevant control variables;
• Critically assess the strength of the proposed design with respect to internal and external validity; be specific about alternative explanations for the outcome rendered less plausible by the design; and
• Identify *real* data sources that would be used in the evaluation.  The actual data need to exist and be accessible to researchers to answer the evaluation question you pose.
  • Provide the source for data. (Data from previous true experiments can serve as supplemental, but not primary, data.)
  • For what years do you have data?
  • At what level of analysis?
  • What variables from the dataset will you use to operationalize your outcomes, controls?
  • Will your data be derived from more than one dataset?
• Cautionary remarks regarding the limitations of design and analysis (e.g., design concerns emphasizing the conditions for causality, analysis concerns such as sample size, measurement reliability and validity, social response bias).

Note: A “good” question for your outcome evaluation includes all the following:

1. One for which data are available to answer the question.
2. One for which previous evaluations exist.  These extant evaluations can deal with the same or related questions.  The preference is for evaluations in peer-reviewed academic journals, GAO reports, or other credible sources.  Evaluations conducted by the program on itself are used only as a last resort.
3. One which is appropriate to the program you are evaluating; and
4. One about which you possess intellectual curiosity.