Hello today’s webinar is about using the

multi-cohort tool for evaluations of post-secondary education interventions.

The multi-cohort tool is designed to support the development of a plan for

the evaluation of a post-secondary education intervention. It can be used to

identify all of the potential student samples from multiple cohorts and time

points that may be included in analyses of intervention impacts. The tool also

supports decision-making about the study samples to be included in analyses based

on the focus of the evaluation’s research questions. Today’s presentation was

prepared by members of the First in the World evaluation technical assistance

team, Anne Wolf and Cristofer Price. I’m Anne Wolf and I’ll be doing the

presentation today. At the end of the webinar you will

understand the large array of potential student samples in multi-cohort

evaluations. Specifically, samples may group students on a number of dimensions,

such as amount of exposure, grade level, and time relative to the start of the

intervention. And you’ll learn how to use the multi-cohort tool to generate all of

the possible samples for your evaluation and identify the samples that will

address the evaluation research questions. You can access the tool online

on the First in the World evaluation technical assistance website or the IES

evaluation technical assistance website. Many education interventions span more

than one semester or year, are implemented with students at multiple

grade levels, are implemented with multiple cohorts of students, and often

collect data at multiple points in time. This creates an array of options for

organizing the sample depending on the research questions that the evaluation

aims to address. The multi-cohort tool helps identify these options, creating

the opportunity for evaluators’ questions about a program’s effects for students

to examine impacts on students with different amounts of exposure, for

example one semester of exposure, one year of exposure, to the intervention, to

examine programs effects for students at different grade levels, for example

freshmen or sophomores or even by semesters, for students in their first

semester of college or second semester of college, and to examine a program’s

effects for students at different points in time relative to the start of the

intervention, so in semester one or semester two or so on. As you’ll see in

the remainder of this presentation, the tool can be used to identify all the

potential samples of students across multiple cohorts and time points and the

tool also supports decision making about which study samples to

include in the analysis and how to group those samples based on the focus of the

evaluation’s research questions. On the next slide we provide an example of how

many samples can be generated from an apparently simple evaluation design.

For example, consider a two-semester intervention for freshmen and

sophomores, which is implemented with three cohorts of students over three

years. Data from this evaluation could be used

to form at least 24 different samples to examine program impacts by exposure

level, for up to two semesters of exposure, grade level for up to two grade

levels, and time up to six semesters since the start of the intervention. With all the potential analyses that can be

conducted, it’s important to determine what research questions are of highest

priority. Then in planning the evaluation, you can be guided by those research

questions in order to narrow the focus and reduce the number of impacts that

will be estimated. There are a couple of reasons why you might want to reduce the

number of impacts that you will estimate. First, it makes sense to present a more

focused examination of the intervention’s impacts on the primary outcomes rather

than to present many different estimates that may be providing consistent

information about program effects. And second, conducting many tests of the

intervention’s impacts increases the probability of finding a statistically

significant effect when it does not exist. This is called type 1 error. In

order to reduce this risk, the What Works Clearinghouse adjusts for multiple

comparisons by setting a more stringent threshold for statistical significance

based on the number of tests being conducted. Therefore, instead of

estimating all the possible intervention impacts, you want to focus on a

well-considered set of tests that will provide evidence about the intervention’s

effectiveness. And finally you can be guided by the

research questions, your primary research questions to identify the corresponding

samples that should be included in the impact analysis to address those

questions. The multi-cohort tool for post-secondary education interventions

is an interactive Excel based tool. It’s intended to help evaluators make

decisions about who to include in impact analyses. It is used for interventions

and evaluations with multiple cohorts, multiple grade levels, and that span

multiple semesters or years. When deciding about the groups of

students to be included in the evaluation samples, it’s helpful to begin

by identifying all the possible groups of students that will experience the

intervention and comparison conditions. The multi-cohort tool generates a

diagram showing all of these possible groups of students based on answers to a

set of questions about the timing and duration of the intervention, about the

evaluation design, the students participating in the study, and the

timing of data collection. Once the diagram is generated by the multi-cohort

tool, the user identifies the groups of students to be included in the

evaluation sample in order to address key research questions. Here we see the

cover page of the tool. When you open the tool you can see that

the Excel spreadsheet has four tabs. The four tabs are the Read Me First tab, the

design questions tab, the matrix tab, and the matrix explainer tab. Using the multi-cohort tool to identify

your study samples involves five steps. Step one is to select the correct version of the tool. There are different versions to reflect the education level of the intervention and the evaluation design. Therefore, the

first step is to select the appropriate version of the multi-cohort tool. There’s

a general version for most designs and a cluster RCT version. Step two, answer a set

of nine questions about the evaluation design. Your responses will guide the

production of the sample diagram or matrix. Step three is to review the

automatically generated matrix based on your answers to the design questions, a

diagram showing all the groups of students that can potentially be

included in the sample will be generated. You can review it and you can return to

the design questions tab to make any corrections if necessary. Step four is to consult the matrix explainer tab to be sure that you

understand the layout of the sample matrix. A general explanation of the

format of the sample diagram is provided on the matrix explainer tab. And step five,

after the matrix has been generated, you can identify the groups of students that

will be included in the sample for each of the impacts that will be estimated.

Samples may be combined to measure outcomes for students with the same

amount of exposure to the intervention, for example two semesters, in the same

grade level, at the end of freshman year, or at the same point relative to the

start of the intervention, for example semester four. We’ll begin with step one,

selecting the correct version of the tool. There are different versions of the

multi-cohort tool depending on the education level of the students served,

post-secondary education or K-12 education. The tools are built

differently to reflect differences in grade levels and in the academic

calendar, so focusing on semesters for post-secondary education or school years

for K-12. We’re focusing on the post-secondary tool today only. The K-12

multi-cohort tool is forthcoming, it’s not yet available. For each education level, post-secondary

or K-12 there are two versions of the tool based on the evaluation design.

There’s a general version, which is appropriate for all designs other than

cluster RCTs. So it’s used for student level assignment RCTs, for QEDs

with assignment at the student or cluster level, or for regression

discontinuity designs. And then there’s also another version for cluster RCTs.

Overall, the two versions are very similar, but there’s one key difference

in the layout of the diagrams produced by each version.

The key difference is that for cluster RCTs, the diagram includes information

about students’ status as joiners or stayers, because clusters of students such

as classes, instructors, or schools are assigned to either the treatment or

comparison condition, some students may join those clusters after initial group

assignment. These students are called joiners, while other students are already

present in the clusters before initial group assignment, and these students are

called stayers. In cluster RCTs, students joining clusters after random assignment

can bias the estimate of the intervention effect, if those who enter

intervention clusters differ systematically from those who enter

comparison clusters. This distinction between stayers and joiners is not

relevant or applicable for other designs, designs other than cluster RCTs.

Therefore, the distinction does not appear on the general version of the

tool, only on the cluster RCT version of the tool. And finally the potential for

bias from entering individuals may be affected by how long after random

assignment students join clusters, namely bias may differ for students who join

soon after the study begins. These students are called

early joiners, or for students who join clusters at a later time point, these

students are called late joiners. For more information about the

WWC standards for cluster RCT designs, you can see the WWC standards handbook version 4.0 on the WWC website.

Now we’ll go over each of the tabs on the multi-cohort tool, beginning with

the read me first tab. On the read me first tab, you can confirm that you’re

using the appropriate version of the multi-cohort tool by answering two

questions about the education level of the intervention: post-secondary or K-12,

and about the evaluation design: cluster RCT or another design. So you’ll begin by answering question

one. You need to indicate the target population of the intervention and

from a drop-down menu you can select either K-12 or post-secondary. Then for

question two you need to indicate the evaluation design from the drop-down

menu. You can select your evaluation design from among the following options:

RCT assignment at the student level, RCT assignment at the cluster level, QED

assignment at either the student or cluster level, or regression

discontinuity design. Based on your responses the tool will indicate whether

or not you’re using the correct version. So here’s an example where the user has

the general version of the tool, the general version of the post-secondary

tool, and has responded to question one that he or she is using the

post-secondary, is evaluating a post-secondary intervention, and then has

indicated the design as a cluster RCT. Therefore the tool has responded to

these questions indicating that the user is not using the correct version,

and then when this statement appears indicating that you are not using the

correct version you’ll be directed to the correct version of the tool for the

education level and evaluation design that you’ve indicated. So in this case

the message back to the user says please use the version for post-secondary

cluster RCTs. When you’re using the right version of the tool you will

receive a message in a green box stating that you are using the correct version.

So in this example the user has indicated that they are evaluating a

post-secondary intervention using a QED design, and because they are answering

these questions in the general version of the post-secondary multi-cohort tool

they have gotten a message back saying this is the correct version.

Next is step two, answer questions about the evaluation design on the design

questions tab. On the design questions tab there are nine questions about the

evaluation design. The same nine questions are asked on both versions of

the multi-cohort tool. For each question the answer is entered into the orange

box either by typing the response directly or by selecting a response from

the drop-down menu. Once the questions are answered a diagram or matrix will be

generated automatically on the matrix tab, which will show all the possible

groups of students that may be included in the sample. To orient you to the

design questions tab I’m showing you what the tab looks like here and showing

questions 1 to 5. We’ll go over the questions on a later slide and complete

them for an example design. And here this is what questions 6 through 9 look like

on the design questions tab. We can use an example design scenario to complete

the questions. So let’s say we’re evaluating a new first-year seminar for

incoming freshmen, which is intended to improve students’ persistence and credit

accumulation. The evaluation will include two cohorts of incoming freshmen who

participate in the intervention or comparison condition, will collect

outcome data on persistence and credit accumulation after each semester of

exposure and for two semesters post intervention, for a total of four

semesters of data collection for each cohort, and the evaluation will collect

baseline data prior to college, for example collecting SAT scores and Pell

Grant eligibility. This same example can be used for either version of the tool.

The questions and the answers are the same even though the matrix that’s

generated differs for the two versions. So here are questions 1 to 5 on the

design questions tab and we can answer them for our example. So the first

question asks, in which academic year will your intervention program

practice or policy begin? We begin by entering 2015 for our example.

You can just type that into the cell, and then from this point forward

all the other questions are completed using a drop-down menu.

I won’t be illustrating how the drop down menu is used in the presentation,

but you can experiment with the tool on your own and you can also provide

different responses to questions and examine the differences in

the matrices that are generated. Next, you indicate in which semester your

intervention will begin, either fall or spring. In this case we’re entering fall

2015 as the beginning of the intervention. The next question asks, how

many separate cohorts of students are in your study? And in our example there are

two. Then, how long in semesters will data be collected during the intervention? In

this example, data will be collected for four semesters for each of the two

cohorts, which is a total of five semesters of data collection. Because the

second cohort begins one semester later than the first, adding an extra semester

of data collection on to the total. Next we answer the question, what is

the earliest grade level a student could begin participating in or receiving the

intervention? And the answer to that in our example is their first semester in

college because it’s a first-year seminar for students in their first and

second semesters and they begin in their first semester. Then the next question

asks, what is the latest grade level that students could begin participating in or

receiving the intervention? And in our example again the answer to that

question is the first semester in college, but there you can imagine

other interventions that may serve students in multiple grade levels,

beginning at different times, different grades. Okay, next we proceed through

questions six through nine and for question six, there are three

different parts. The first part asks, is baseline measured at the same grade

level for all students? So in our example, we answer yes and we

indicate the grade level in response to questions 6b which says, if you answered

yes for 6a, what grade level is baseline measured for all students? In our example

that’s pre-college. If we had answered no to question 6a and said that

baseline is not measured at the same grade level for all students,

then question 6b would have disappeared and we would respond to questions 6c,

which would have asked, what is the grade level when baseline is measured? And we

would have selected from a drop-down menu options, such as the start of the

first semester of the intervention or one semester before the intervention and

the like, but because we’ve answered yes, then

question 6c has disappeared from our from our options. Next we go on to

question 7, which asks, what is the maximum number of semesters that

students can participate in the intervention or comparison condition?

Which is two semesters in our example. Question 8 asks, what is the highest

grade level during which students can participate in the intervention or

comparison condition? Which is their second semester in college in our

example. We have a two semester, one year intervention for

students that they participate in during their first semester in college and

their second semester in college. And then the last question asks, how many

semesters post-intervention will you collect data?

And in our example, as we’ve stated that’s two semesters post-intervention.

Okay next we can review the automatically generated matrix

on the matrix tab. Once all the design questions have been

answered, the matrix tab will provide a customized diagram that shows all the

groups of students that can potentially be included in the sample for impact

analyses. We’re going to look at the two versions of the matrices produced from

our example, one at a time. First we’ll look at the matrix produced by the

general version of the tool and then we’ll look at the matrix produced by the

cluster RCT version. So this is the matrix produced from the general version

of the tool. The layout of the rows is the same in both versions of the

matrix. The top row, which is shaded beige, provides a letter to label each of the

columns that have information about students in the sample. This enables the

user to refer to specific cells by naming the column letter and the row

number, for example cell a1 or cell a2. Then there’s the cohort row, which

provides the cohort number. Each cohort is shaded a different color. The shaded

color may vary depending on the number of cohorts in your evaluation. In this

example, cohort 1 is shaded blue and cohort 2 is shaded purple. Next is the

baseline row, which provides information about when the baseline variable is

measured for students shown in the same column. The baseline cells are shaded

orange and in our example baseline is measured before college so the cells say

pre-college for each column and each cohort. Okay next are the numbered rows and

these rows correspond to the progression of time in semesters relative to the

start of the intervention. Each row represents a semester, beginning with the

first semester that the intervention is implemented during which data will be

collected. In this example as well as in the cluster RCT example that we’ll be

looking at there five rows. Row one is semester one, which is the fall

semester of the 2015-2016 school year in our example. Row 2 is semester two, the

spring semester of the same academic year, spring 2016.

Row three is semester three, row four is semester four, and row five is semester five. So fall 16, spring 17, and fall 2017. In the general version of the matrix, the

columns correspond to each grade level within a cohort. In this example there

are two cohorts, each with students from one grade level, first semester freshman. Therefore, there

are a total of two lettered columns. Column A shows the progression of

incoming freshmen in cohort one from their first semester in college through

their fourth, and column B shows the progression of incoming freshmen in

cohort two. The cells in the columns include information about two aspects of

time. Grade-level, the cells show students

grade level in college in semesters as students move from

their first semester to their second semester to their third semester through

their fourth semester in college. And the cells also include information about the

amount of exposure, the number of semesters of exposure to the

intervention which depends on the duration of the intervention. So one semester of exposure to the

intervention or comparison condition is shaded red and occurs in fall 2015 for

cohort one in our example, and spring 2016 for cohort two in our example. Two

semesters of exposure is shaded green. And then, in addition to information

about the amount of exposure cells may also include information about the

number of semesters since the end of the intervention if data collection

continues past the end of the intervention. So in the example, data are

collected one semester post intervention, which is written as 1S post referring

to one semester post intervention and two semesters post intervention is

written as 2S post, And note that as described above, the

third aspect of time captured by the matrix in addition to grade level and

amount of exposure or post-intervention follow-up, is the time relative to the

start of the intervention, which as we mentioned before is represented by the

rows in the diagram. Okay now here is a the matrix produced by the

cluster RCT version of the tool. Assuming that we answered the nine

questions in the design question tab in just the same way that we did for the

general version of the tool. Okay and again remember the example is

an intervention that begins in fall of the 2015-2016 school year with data

collected for a total of five semesters, from two cohorts of first semester

students, who participate in a two semester intervention, with outcomes

collected for two semesters post-intervention. In the cluster RCT

version of the matrix, the layout of the rows is the same as in the general

version. The top row is shaded beige and provides a letter to label the columns.

The second row provides the cohort number, the third row provides

information about when baseline is measured for students shown in the same

column. The next set of rows corresponds to the progression of time relative to

the start of the intervention, with each row representing a semester. Again there

are five rows, one for each semester of data collection across the two cohorts.

In the matrix for cluster RCTs, just as in the general version of the tool, the

lettered columns also correspond to each grade level within a cohort of students.

However for each grade level in the first cohort of students, the columns are

divided into three parts distinguishing students’ status as stayers, early joiners,

or late joiners. In this example, there are two cohorts of students.

Each with students from one grade level, first semester freshmen. The first cohort

of students, instead of having one column of students in their first semester in

college, there are three columns. Column A shows the progression of incoming

freshman and cohort one who joined the cluster well after random assignment,

these are the late joiners. Column B shows the progression of incoming

freshmen in cohort one who joined the cluster soon after random assignment,

these are the early joiners. And column C shows the progression of incoming

freshmen in cohort one who were present in clusters at

the time of random assignment and are labeled stayers. All subsequent cohorts of

students are late joiners, because the students enter the clusters well after

random assignment, at least one semester later. In this example the second cohort

of first semester freshmen appears in column D and the cell number D2 is

labeled late joiners. Apart from distinguishing students’ status of stayers,

early joiners, or late joiners, the layout of the cluster RCT matrix is the same as

the general version of the tool. In the cluster RCT version, the cells in each

column include the same information previously described for the general

version about grade level and about amount of exposure to the intervention

or post intervention follow-up. Next is step 4, to consult the matrix

explainer tab. You want to do that to be sure you understand the layout of the

sample matrix. The matrix explainer tab provides illustrations and

definitions for each part of the diagram produced in the matrix tab. For

orientation we show what the matrix explainer tab looks like on the general

version of the multi-cohort tool. There are explanations for three aspects of

the diagram that are provided as a reference for the user. The first two

statements and illustrations focus on the baseline row and the cohort row, and

those are the same in both versions of the tool. But, the third statement and

illustration, which is about the remaining rows, showing the progression

of time that differs on the two versions. So we’re going to look at the

third statement for both versions and they’ll be described in this section. So

on the next set of slides we’ll look more closely at each of these three

statements and illustrations. The first statement shown on the matrix

explainer tab again is the same for both versions of the tool. It describes and

illustrates the cohorts row in the matrix. For each cohort, cells

are shaded a different color. In the example shown on the matrix explainer

tab, there are three cohorts labeled one for cohort one and shaded blue, two for

cohort two and shaded white, and three for cohort three shaded purple. Note that

there are four columns for each cohort, and as described previously, each

column corresponds to a grade level within a cohort. Therefore, four columns

per cohort occur when there are four grade levels in each cohort. The cohort

row does not provide information about grade level. Grade level information

appears on a later row in the matrix and is described in the third

illustration on the matrix explainer tab. The second statement shown on the matrix explainer tab is also the same in both versions of the tool. It describes and

illustrates the baseline row of the matrix. In the matrix baseline cells are

all shaded orange and the values are the grade level when baseline is measured. In

the example shown, baseline is measured at the beginning of the first treatment

or comparison semester, therefore the grade level differs for

each column depending on students’ grade level at entry into the study: their

first semester in college, second semester in college, third or fourth

semester in college. In the general version of the tool, the third statement

on the matrix explainer tab describes and illustrates the remaining rows of

the matrix. Here I’m only able to show the first sentence that appears on the

tab just to keep the text on the slide on the screen or readable size, but I

will go over all the information about the rows that’s covered on the matrix

explainer tab. So all subsequent rows have a heading to indicate the school

year and the semester, fall or spring, that’s represented by the row. And as

described previously, cells in these rows provide information about students’ grade

level and amount of exposure to the condition or the post-intervention

follow-up. Okay grade level, which is semesters

in college, is written as text without any shading. For example first semester

in college is written as first semester, second semester, third semester, and so on. Semesters of exposure to the intervention or comparison condition are

shown as shaded cells. In this example one semester of exposure is written as

one and shaded red, two semesters of exposure, written as two, is shaded yellow,

and three semesters of exposure, written as three, is shaded green. If data

collection continues past the end of the intervention, the number of

post-intervention semesters is indicated. One semester post intervention is

written as 1S post without shading. In the cluster RCT version of the tool,

the third statement shown on the matrix explainer tab also describes and

illustrates the remaining rows of the matrix. Again all the text that appears

on the matrix explainer tab is not shown here for space reasons, but I’ll go over

all the information. The rows follow the same structure as the general version,

with row headings that indicate the school year and semester, fall or spring.

And cells in these rows provide information about grade level and amount

of intervention exposure or post- intervention follow-up. The difference

between the cluster RCT and the general version is that the cells in the cluster

RCT version of the tool also distinguished students as stayers, early

joiners, or late joiners based on when they joined the cluster relative to the

timing of random assignment. Okay once the sample diagram or matrix is generated, the next step is to

identify the specific groups of students from each relevant cell in the matrix

that will be included in the evaluation sample used to address each research

question for each impact that will be estimated.

So impacts could be estimated for students with the same amount of

exposure to the intervention or comparison condition regardless of their

grade level or the point in time relative to the start of the

intervention. Or, impacts could be estimated for students at the same grade

level regardless of the amount of exposure to the intervention or time

relative to the start of the intervention. Or, impacts could be

estimated for students at the same point in time relative to the start of the

intervention regardless of their amount of exposure to the intervention or their

grade level. In order to identify the cells in the

matrix that will be included in the evaluation sample, you can create a

version of the matrix that can be copied or edited. To do this you click the

button on the matrix tab that says generate editable version of the matrix and a new spreadsheet will be created with a copy of the matrix. The

new version of the matrix can be edited and it can be copied and pasted into

other documents such as an evaluation plan. And then you can highlight the

cells in the matrix representing the students to be included in the impact

analyses for each research question or you can simply reference the relevant

cell numbers using the column letters and row numbers. I want to point out that

the multi-cohort tool for post-secondary interventions generates diagrams in

which academic terms are represented as semesters. For post-secondary

interventions that involve quarters or trimesters or other terms, it will be

necessary for you to modify the matrix to accurately reflect the actual

academic terms in your evaluation. Okay we’re going to go over examples for

how to use the matrix to identify the student sample for three different

research questions, focused on amount of exposure grade level or time point

relative to the intervention. So for amount of exposure, we can consider the

research question, “what is the impact of the intervention on persistence and

credit accumulation for students after one semester of the intervention, one

semester of exposure?” For grade level, we might focus on a research question like,

“what is the impact of the intervention on students at the end of their

sophomore year or their fourth semester in college?” And for time since the start

of the intervention, we could consider the research question, “what is the impact

of the intervention on students after the intervention has been in colleges

for three years at the end of semester six?” These examples are presented for

both versions of the matrix. First, we’re going to go through these three examples

for the general version and then we’ll go through the same three examples for

the cluster RCT version. Before we go through the examples of how to identify

the samples based on the research questions, I’ll introduce the matrix

showing all the potential samples, which will be the basis of our example

research questions. So this is a matrix generated by the

general version of the multi-cohort tool. The matrix would be produced

for a student RCT, a QED, or regression discontinuity design, but not a cluster

RCT. For a cluster RCT, you would use the other version of the tool. And, as I

mentioned will look at examples using the cluster RCT after we look at

examples using the general version. In this matrix there are three cohorts of

students who participate in the intervention or comparison condition, for

up to two semesters in their first and second semester in college or who

participate in the business as usual condition. During that time, our

outcomes will be measured each semester from fall 2015 through spring 2016. We begin with example one, ‘what is the

impact of the intervention on persistence and credit accumulation for

students after one semester of exposure to the intervention?” For this research

question, the student sample would be comprised of students in cells

representing one semester of exposure regardless of their grade level and

regardless of the program semester, the time since the start of the intervention.

In the example matrix, outcomes are measured for students in six cells shown

here outlined in red. These students have the same amount of exposure to the

intervention. All students will have had one semester of exposure to their

assigned condition when outcomes are measured. The students in the sample will

be different grade levels when outcomes are measured. Some will be in their first

semester in college and others will be in their second semester in college, and

students will have outcomes measured at different times relative to the start of

the intervention. Some students will have outcomes measured in semester one fall

2015 in this example, some in semester two, and

some in semester three. Similarly, samples could be identified to

examine outcomes for students with two semesters of exposure, which are shown here shaded green, and data would come from these three

cells, three groups of students. Or, outcomes could be measured for students

at two semesters post-intervention. In these examples, students are the same

grade level, but outcomes are measured at different time points relative to the

start of the intervention. Next we’ll look at example two, “what is the impact

of the intervention on students at the end of their sophomore year, their fourth

semester in college?” For this research question, the student sample would be

comprised of all the cells in the matrix in which students are sophomores,

regardless of their amount of exposure or the amount of time since the start of

the intervention. In this example the the sample would

include students in the six cells that are outlined in red. Students in the

sample would all be the same grade level, they would all be in their fourth

semester in college. The students in the sample would have different amounts of

exposure to the intervention. Some students will have one semester of

exposure and others will have two semesters of exposure. And students in the sample will have

outcomes measured at different times relative to the start of the

intervention. For some students outcomes will be measured in

semester three, for some in semester four, for some in semester five, and for some

in semester six. Similarly, samples could be identified to

examine outcomes for students at the end of their junior year, at the end of their

sixth semester in college, as shown here. Next is example three with the research

question, “what is the impact of the intervention on students after the

intervention has been in colleges for three years, at the end of semester six?”

For this research question, the student sample would be comprised of all cells

in the matrix corresponding to semester six, regardless of student’s amount of

exposure to the intervention or their grade level. In the example matrix, the

sample would include students in five cells shown here outlined in red.

Outcomes would be measured at the same time relative to the start of the

intervention. All students in the sample in semester

six will have outcomes measured. Students in the sample would have

different amounts of exposure to the invention. Some students will have one

semester of exposure and others will have two semesters of exposure. And

outcomes will be measured at different post-intervention follow-up time points.

Some students will have outcomes measured four semesters post

intervention, some three semesters post intervention, and some at two semesters

post intervention. And students in the sample would be at different grade

levels. Some students would be in their sixth semester in college, some in

their fifth semester, and some in their fourth semester in college. Okay now we can go through these same

examples again, but this time using the cluster RCT version of the tool.

So again we can look at three different research questions. One focused

on impacts after one semester of exposure, one focused on impacts on

students at the end of their sophomore year, the end of their fourth semester in

college, and one focused on impacts of the intervention after it has been in

colleges for three years, at the end of semester six. But before we go through

the examples for the cluster RCT version, I want to introduce the matrix generated

by the cluster RCT version of the tool, which forms the basis for our example

research questions. So it’s the same intervention

and design as we described for the general version, but the matrix looks

different. In this cluster RCT, there’s one cohort of schools that’s randomly

assigned to the treatment or comparison condition, and within those schools there

are three cohorts of students who participate in the intervention for up

to two semesters in college, or who participate in the business-as-usual

comparison condition during that time. And outcomes will be measured for each

semester from fall 2015 through spring 2018. With a cluster RCT design again

it’s necessary to specify whether the sample will include students who were

present in clusters at the time of random assignment, stayers,or whether the

sample will also include students who joined clusters soon after random

assignment. In other words, whether this sample will include stayers and early

joiners, or whether the sample will include any students who are in clusters

at the time that outcomes are measured. In other words, that the sample will

include stayers, early joiners, and late joiners. So returning to example one for

the cluster RCT design, we look at the samples that would be used to address

the research question of, “what is the impact of the intervention on

persistence and credit accumulation for students after one semester of exposure

to the intervention?” For this question, for the cluster RCT again we need to

determine whether the sample will include joiners or not. If we were to

focus on the sample of students that have one semester of exposure regardless

of grade level or semester since the start of the intervention, and focus on

students who were present in schools at the time of random assignment, a sample

with only stayers, then outcomes would be measured for students in two cells shown

here outlined in red. Specifically, students in the sample

would have the same amount of exposure to the intervention. All students would

have had one semester of exposure and all students would be stayers, students

who were present in schools or whatever the clusters are, at the time of random

assignment. Students’ outcomes would be measured at the same time relative to

the start of the intervention for all students. Outcomes would be measured in

semester one, but for students at different grade levels the sample would

include some students in their first semester in college and others in their

second semester. Similarly, samples could be identified to

include early and late joiners as well as stayers when examining outcomes for

students with one semester of exposure. So you could expand to include joiners

and many more groups of students would be included, including those from cohorts

2 & 3. Or if you wanted to examine impacts on students with two semesters

of exposure and includes stayers, early joiners, and late joiners, then the

sample would include students in five cells, the five cells with a green bar

showing two semesters of exposure and these cells are shown outlined in pink. Next we’ll look at example 2, the

question about impacts on students at the end of their sophomore year or the

end of their fourth semester in college. The sample will include students at the

end of their sophomore year, fourth semester in college, regardless of their amount of

exposure to the intervention or the time since the start of the intervention. And

then if the impact analysis focuses on sophomores who were present in schools

at the time of random assignment and soon after, in other words both stayers

and early joiners, then outcomes would be measured for students shown in four

cells, those cells that are outlined in red. Specifically, students in the sample

would be the same grade level, all students would have completed their

fourth semester in college. The sample would include students

present in schools at the time of random assignment or soon after, stayers and

early joiners. Students would have had, in the sample, would have had different

amounts of exposure to the intervention. Some with one semester of exposure and

others with two semesters of exposure. For all, then for all students

outcomes will be measured two semesters post-intervention and outcomes will be

measured at different time relative to the start of the intervention. For some

students, outcomes will be measured in semester three and for other students in

semester four. Similarly, samples could be identified to

examine outcomes for students at the end of their junior year, their sixth

semester in college, including stayers and early joiners, which are shown here

outlined in pink. Last is example three for the cluster RCT design, addressing

the research question, “what is the impact of the intervention on students after

the intervention has been in colleges for three years?” Measuring

outcomes at the end of semester six for the cluster RCT design. Again we need to

just specify whether the sample will include stayers, early joiners, and or

late joiners. If the impact analysis focuses on all students in

semester six, regardless of the amount of exposure or grade level, and regardless

of when they entered schools, in other words a sample that includes stayers,

early joiners, and late joiners, then outcomes would be measured for students

in five cells shown here outlined in red. Specifically, outcomes would be measured

for students at the same time relative to the start of the intervention.

Outcomes would be measured at the end of semester six. Students in the sample

would include those who were present in schools at the… all students who were

present in schools at the end of semester six, regardless of when they

entered the schools, whether they were stayers, early joiners, or late joiners.

Outcomes would be measured for students at different grade levels. Some students

in the sample would have completed their fourth semester in college, others would

be in their fifth semester in college, and others would be in their sixth

semester in college. Outcomes would be measured at different time points post intervention. Outcomes would be measured two semesters post intervention for some

students, three semesters post intervention for some students, and four

semesters post intervention for some students. All

students would have had two semesters of exposure to the intervention. Okay and that’s all three

examples for both versions of the tool. So from this webinar you should know how

to select the correct version of the multi-cohort tool for post-secondary

education interventions for your evaluation. You should be able to use it

to generate a customized matrix for your evaluation design that allows you to

identify the samples to include in your impact analysis based on the focus of

the evaluation research questions. You may find it helpful to revisit the tool

and your customized matrix over the course of the evaluation, especially if

changes occur in the intervention or your evaluation plan. I want to point you

to some additional resources for the multi-cohort tool. There’s of course the

multi-cohort tool for post secondary interventions itself, the tool itself.

Both the general version and the cluster RCT version. And there’s also a user

guide. All of these things are available online on the First in the

World evaluation technical assistance website or the IES evaluation technical

assistance website. And finally if you’re conducting an evaluation that’s focused

on an intervention serving students in Kindergarten through grade 12, there is

an analogous tool, K-12 multi-cohort tool and user guide, which are forthcoming.

They’re not available yet. And finally the recording of this webinar is

also available online on the First in the World evaluation technical

assistance website and the IES evaluation technical assistance website.

Thank you.