Presenter:
Mike Trevisan
Dean and Professor
Washington State University

Tamara Walser
Professor
University of North Carolina-Wilmington


Description:
The purpose of this workshop is to explore and engage the transformative use and potential of evaluability assessment (EA). EA was developed in the 1970s as a pre-evaluation activity for determining if a program was ready for outcome evaluation, with a focus on management as primary intended users. Much like evaluation in general, EA theory and practice have evolved to address the complex needs of programs and their communities. No longer tied exclusively to management decisions about outcome evaluation, EA can be used as a collaborative evaluation approach at any point in a program's lifecycle. Transforming our understanding and application of EA unlocks its potential to engage program and organization communities in evaluation, address program complexity, support culturally responsive and equity-focused evaluation, and build evaluation capacity. Through examples, case scenarios, small group applied learning activities, and discussion; workshop participants will consider and apply a transformative EA approach. Specifically, they will explore current theory and uses of EA, develop skills to implement transformative EA, and consider ways to implement transformative EA in their practice. This workshop is at the intermediate level. Participants may be new to EA; however, they should have foundational knowledge and skills in evaluation. Although EA is still used to inform management decisions of readiness for outcome evaluation, transformative EA is a collaborative approach that can be used at any point in a program's lifecycle, including as ongoing participatory EA (Thurston & Potvin, 2003). Because it is no longer tied exclusively to outcome evaluation, EA supports recommendations regarding the feasibility and best approaches for further evaluation given program context and culture, and broader organization and community needs. Engaging a collaborative EA process to develop or revisit an initial program theory model; gather feedback data on the model to assess its meaningfulness, plausibility, and utility; and make recommendations for the program, as well as further or ongoing evaluation has transformative potential. For example, it can better address program complexity, support culturally responsive and equity-focused evaluation, and build evaluation capacity.


Learning Outcomes:

• Current theory and uses of evaluability assessment
• How to implement an evaluability assessment
• How evaluability assessment can support and enhance meaningful stakeholder engagement, culturally responsive evaluation, and build evaluation capacity.

Agenda and Learning Strategies:
Introductions and Workshop Overview (15 minutes)
Facilitator Introductions
Overview of Learning Outcomes
Workshop Agenda and Format Participant Introductions and Workshop Goals (Think-Pair-Share Activity)
Think-Pair-Share Activity: Participants will reflect and identify their individual workshop goals, then will be paired in breakout rooms to share introductions and goals. When back together as a group, as time permits, pairs can volunteer to share their introductions and goals with the whole group. The purpose of the activity is for participants to get to know each other and to articulate their workshop goals.

Current Theory and Uses of Evaluability Assessment (45 minutes)
Current Evaluability Assessment Theory and Practice
Resurgence of Evaluability Assessment across Disciplines and Globally Use of Evaluability Assessment to Effectively Engage Stakeholders (Chalk Talk Activity Part One) Chalk Talk Activity Part One: Facilitators will present overview definitions and resources related to stakeholder engagement. Each of the following questions will be shared in an online, collaborative Padlet (see padlet.com): How can evaluability assessment be used for stakeholder engagement? Culturally responsive evaluation? Evaluation capacity? How will stakeholders and their information needs be identified through the evaluability assessment? How can evaluability assessment be integrated into my work? Participants will add their responses to the Padlet during the workshop
we will ask them to add responses as they come to mind during the workshop and will also prompt them at key points to add responses, comment on others' responses, and "star" responses that resonate with them.

The Basics of Implementing an Evaluability Assessment (60 minutes)
1. Focusing the Evaluability Assessment (Small Group Activity)
2. Developing an Initial Program Theory (Small Group Activity)
3. Gathering Feedback on Program Theory (Small Group Activity)
4. Using the Evaluability Assessment (Small Group Activity) Small Group Activity: Participants will be walked through each of the four components of the evaluability assessment model by responding, in small groups, to a series of guiding questions applied to evaluability assessment case scenarios. Whole-group discussion will be used to determine common themes across group responses and reflect on implications for practice. The purpose of the case scenarios is to build knowledge, skills, and dispositions needed for successful evaluability assessment practice. We will facilitate activities in the virtual environment through the use of breakout rooms for small groups; the chat function; and collaborative tools, such as Padlet, Miro, and Google docs.

Current Theory and Uses of Evaluability Assessment Continued (30 minutes)
Use of Evaluability Assessment for Stakeholder Engagement (Chalk Talk Activity Part Two) Chalk Talk Activity Part Two: Participants will be given time to review the Padlet questions and make any final additions. Then the facilitators will lead a discussion on learnings from the activity and implications for using evaluability assessment to effectively engage stakeholders.

Final Questions and Wrap-Up (15 minutes) AEA Mid-workshop Break (15 minutes)

NOTE: All times can be adjusted as needed


This workshop is aligned to AEA’s Competencies and Guiding Principles as follows:
EA requires strong methods skills, interpersonal skills, integrity, attention to detail, and a focus on equity. Each of the four EA components have guiding questions that focus on the Guiding Principles. In addition, as participants work through case scenarios, they will be asked to address the Guiding Principles. Thus, the workshop has strong alignment with the Guiding Principles.


Note that the same features expressed for the Guiding Principles are also present for AEA Evaluator Competencies. In short, there is strong alignment between the workshop and AEA Evaluator Competencies through all workshop activities.


Facilitation Experience:
We have facilitated several professional development workshops on evaluability assessment including workshops for the Claremont Evaluation Center Professional Development Workshop series, European Evaluation Society biennial conference, Canadian Evaluation Society annual conference, United Nations Population Fund (3-day training), American Evaluation Association annual conference, American Evaluation Association Summer Evaluation Institute, Frontiers in Education annual conference (Madrid, Spain), and Khon Kaen University (Khon Kaen, Thailand). Audiences have included practitioners, graduate students, and university faculty; as well as participants from around the world. The feedback we have received has been largely positive, with many participants noting the relevance and usefulness of what they learned. We have made changes to our materials and learning strategies based on feedback and reflection to increase engagement and application. We have taught 3-hour, 6-hour, and 3-day workshops on evaluability assessment and through these experiences have been able to tailor the format and content to fit these various timeframes. As university faculty, our area of focus for teaching and research is evaluation. Our teaching has included program evaluation, research methods, assessment, and statistics. We have both conducted numerous evaluability assessments within local school districts, higher education, and community organizations. We also participated as technical advisors for a national evaluability assessment sponsored by the federal government. Our writing includes numerous peer-reviewed journal articles on the use of evaluability assessment, including a 20-year review of evaluability assessment practice found in the peer-reviewed literature. The article was published in the American Journal of Evaluation (2007), and has been frequently cited in journal articles on evaluability assessment. We also wrote the most recent book on evaluability assessment, Evaluability Assessment: Improving Evaluation Quality and Use (2015, Sage). Collectively, we have over 50 years of experience working as evaluators and evaluation educators.


Date:
July 7th, 12:00 PM - 3:00 PM Eastern Time