Using the Multi-Cohort Tool for Postsecondary Interventions


Hello today’s webinar is about using the
multi-cohort tool for evaluations of post-secondary education interventions.
The multi-cohort tool is designed to support the development of a plan for
the evaluation of a post-secondary education intervention. It can be used to
identify all of the potential student samples from multiple cohorts and time
points that may be included in analyses of intervention impacts. The tool also
supports decision-making about the study samples to be included in analyses based
on the focus of the evaluation’s research questions. Today’s presentation was
prepared by members of the First in the World evaluation technical assistance
team, Anne Wolf and Cristofer Price. I’m Anne Wolf and I’ll be doing the
presentation today. At the end of the webinar you will
understand the large array of potential student samples in multi-cohort
evaluations. Specifically, samples may group students on a number of dimensions,
such as amount of exposure, grade level, and time relative to the start of the
intervention. And you’ll learn how to use the multi-cohort tool to generate all of
the possible samples for your evaluation and identify the samples that will
address the evaluation research questions. You can access the tool online
on the First in the World evaluation technical assistance website or the IES
evaluation technical assistance website. Many education interventions span more
than one semester or year, are implemented with students at multiple
grade levels, are implemented with multiple cohorts of students, and often
collect data at multiple points in time. This creates an array of options for
organizing the sample depending on the research questions that the evaluation
aims to address. The multi-cohort tool helps identify these options, creating
the opportunity for evaluators’ questions about a program’s effects for students
to examine impacts on students with different amounts of exposure, for
example one semester of exposure, one year of exposure, to the intervention, to
examine programs effects for students at different grade levels, for example
freshmen or sophomores or even by semesters, for students in their first
semester of college or second semester of college, and to examine a program’s
effects for students at different points in time relative to the start of the
intervention, so in semester one or semester two or so on. As you’ll see in
the remainder of this presentation, the tool can be used to identify all the
potential samples of students across multiple cohorts and time points and the
tool also supports decision making about which study samples to
include in the analysis and how to group those samples based on the focus of the
evaluation’s research questions. On the next slide we provide an example of how
many samples can be generated from an apparently simple evaluation design.
For example, consider a two-semester intervention for freshmen and
sophomores, which is implemented with three cohorts of students over three
years. Data from this evaluation could be used
to form at least 24 different samples to examine program impacts by exposure
level, for up to two semesters of exposure, grade level for up to two grade
levels, and time up to six semesters since the start of the intervention. With all the potential analyses that can be
conducted, it’s important to determine what research questions are of highest
priority. Then in planning the evaluation, you can be guided by those research
questions in order to narrow the focus and reduce the number of impacts that
will be estimated. There are a couple of reasons why you might want to reduce the
number of impacts that you will estimate. First, it makes sense to present a more
focused examination of the intervention’s impacts on the primary outcomes rather
than to present many different estimates that may be providing consistent
information about program effects. And second, conducting many tests of the
intervention’s impacts increases the probability of finding a statistically
significant effect when it does not exist. This is called type 1 error. In
order to reduce this risk, the What Works Clearinghouse adjusts for multiple
comparisons by setting a more stringent threshold for statistical significance
based on the number of tests being conducted. Therefore, instead of
estimating all the possible intervention impacts, you want to focus on a
well-considered set of tests that will provide evidence about the intervention’s
effectiveness. And finally you can be guided by the
research questions, your primary research questions to identify the corresponding
samples that should be included in the impact analysis to address those
questions. The multi-cohort tool for post-secondary education interventions
is an interactive Excel based tool. It’s intended to help evaluators make
decisions about who to include in impact analyses. It is used for interventions
and evaluations with multiple cohorts, multiple grade levels, and that span
multiple semesters or years. When deciding about the groups of
students to be included in the evaluation samples, it’s helpful to begin
by identifying all the possible groups of students that will experience the
intervention and comparison conditions. The multi-cohort tool generates a
diagram showing all of these possible groups of students based on answers to a
set of questions about the timing and duration of the intervention, about the
evaluation design, the students participating in the study, and the
timing of data collection. Once the diagram is generated by the multi-cohort
tool, the user identifies the groups of students to be included in the
evaluation sample in order to address key research questions. Here we see the
cover page of the tool. When you open the tool you can see that
the Excel spreadsheet has four tabs. The four tabs are the Read Me First tab, the
design questions tab, the matrix tab, and the matrix explainer tab. Using the multi-cohort tool to identify
your study samples involves five steps. Step one is to select the correct version of the tool. There are different versions to reflect the education level of the intervention and the evaluation design. Therefore, the
first step is to select the appropriate version of the multi-cohort tool. There’s
a general version for most designs and a cluster RCT version. Step two, answer a set
of nine questions about the evaluation design. Your responses will guide the
production of the sample diagram or matrix. Step three is to review the
automatically generated matrix based on your answers to the design questions, a
diagram showing all the groups of students that can potentially be
included in the sample will be generated. You can review it and you can return to
the design questions tab to make any corrections if necessary. Step four is to consult the matrix explainer tab to be sure that you
understand the layout of the sample matrix. A general explanation of the
format of the sample diagram is provided on the matrix explainer tab. And step five,
after the matrix has been generated, you can identify the groups of students that
will be included in the sample for each of the impacts that will be estimated.
Samples may be combined to measure outcomes for students with the same
amount of exposure to the intervention, for example two semesters, in the same
grade level, at the end of freshman year, or at the same point relative to the
start of the intervention, for example semester four. We’ll begin with step one,
selecting the correct version of the tool. There are different versions of the
multi-cohort tool depending on the education level of the students served,
post-secondary education or K-12 education. The tools are built
differently to reflect differences in grade levels and in the academic
calendar, so focusing on semesters for post-secondary education or school years
for K-12. We’re focusing on the post-secondary tool today only. The K-12
multi-cohort tool is forthcoming, it’s not yet available. For each education level, post-secondary
or K-12 there are two versions of the tool based on the evaluation design.
There’s a general version, which is appropriate for all designs other than
cluster RCTs. So it’s used for student level assignment RCTs, for QEDs
with assignment at the student or cluster level, or for regression
discontinuity designs. And then there’s also another version for cluster RCTs.
Overall, the two versions are very similar, but there’s one key difference
in the layout of the diagrams produced by each version.
The key difference is that for cluster RCTs, the diagram includes information
about students’ status as joiners or stayers, because clusters of students such
as classes, instructors, or schools are assigned to either the treatment or
comparison condition, some students may join those clusters after initial group
assignment. These students are called joiners, while other students are already
present in the clusters before initial group assignment, and these students are
called stayers. In cluster RCTs, students joining clusters after random assignment
can bias the estimate of the intervention effect, if those who enter
intervention clusters differ systematically from those who enter
comparison clusters. This distinction between stayers and joiners is not
relevant or applicable for other designs, designs other than cluster RCTs.
Therefore, the distinction does not appear on the general version of the
tool, only on the cluster RCT version of the tool. And finally the potential for
bias from entering individuals may be affected by how long after random
assignment students join clusters, namely bias may differ for students who join
soon after the study begins. These students are called
early joiners, or for students who join clusters at a later time point, these
students are called late joiners. For more information about the
WWC standards for cluster RCT designs, you can see the WWC standards handbook version 4.0 on the WWC website.
Now we’ll go over each of the tabs on the multi-cohort tool, beginning with
the read me first tab. On the read me first tab, you can confirm that you’re
using the appropriate version of the multi-cohort tool by answering two
questions about the education level of the intervention: post-secondary or K-12,
and about the evaluation design: cluster RCT or another design. So you’ll begin by answering question
one. You need to indicate the target population of the intervention and
from a drop-down menu you can select either K-12 or post-secondary. Then for
question two you need to indicate the evaluation design from the drop-down
menu. You can select your evaluation design from among the following options:
RCT assignment at the student level, RCT assignment at the cluster level, QED
assignment at either the student or cluster level, or regression
discontinuity design. Based on your responses the tool will indicate whether
or not you’re using the correct version. So here’s an example where the user has
the general version of the tool, the general version of the post-secondary
tool, and has responded to question one that he or she is using the
post-secondary, is evaluating a post-secondary intervention, and then has
indicated the design as a cluster RCT. Therefore the tool has responded to
these questions indicating that the user is not using the correct version,
and then when this statement appears indicating that you are not using the
correct version you’ll be directed to the correct version of the tool for the
education level and evaluation design that you’ve indicated. So in this case
the message back to the user says please use the version for post-secondary
cluster RCTs. When you’re using the right version of the tool you will
receive a message in a green box stating that you are using the correct version.
So in this example the user has indicated that they are evaluating a
post-secondary intervention using a QED design, and because they are answering
these questions in the general version of the post-secondary multi-cohort tool
they have gotten a message back saying this is the correct version.
Next is step two, answer questions about the evaluation design on the design
questions tab. On the design questions tab there are nine questions about the
evaluation design. The same nine questions are asked on both versions of
the multi-cohort tool. For each question the answer is entered into the orange
box either by typing the response directly or by selecting a response from
the drop-down menu. Once the questions are answered a diagram or matrix will be
generated automatically on the matrix tab, which will show all the possible
groups of students that may be included in the sample. To orient you to the
design questions tab I’m showing you what the tab looks like here and showing
questions 1 to 5. We’ll go over the questions on a later slide and complete
them for an example design. And here this is what questions 6 through 9 look like
on the design questions tab. We can use an example design scenario to complete
the questions. So let’s say we’re evaluating a new first-year seminar for
incoming freshmen, which is intended to improve students’ persistence and credit
accumulation. The evaluation will include two cohorts of incoming freshmen who
participate in the intervention or comparison condition, will collect
outcome data on persistence and credit accumulation after each semester of
exposure and for two semesters post intervention, for a total of four
semesters of data collection for each cohort, and the evaluation will collect
baseline data prior to college, for example collecting SAT scores and Pell
Grant eligibility. This same example can be used for either version of the tool.
The questions and the answers are the same even though the matrix that’s
generated differs for the two versions. So here are questions 1 to 5 on the
design questions tab and we can answer them for our example. So the first
question asks, in which academic year will your intervention program
practice or policy begin? We begin by entering 2015 for our example.
You can just type that into the cell, and then from this point forward
all the other questions are completed using a drop-down menu.
I won’t be illustrating how the drop down menu is used in the presentation,
but you can experiment with the tool on your own and you can also provide
different responses to questions and examine the differences in
the matrices that are generated. Next, you indicate in which semester your
intervention will begin, either fall or spring. In this case we’re entering fall
2015 as the beginning of the intervention. The next question asks, how
many separate cohorts of students are in your study? And in our example there are
two. Then, how long in semesters will data be collected during the intervention? In
this example, data will be collected for four semesters for each of the two
cohorts, which is a total of five semesters of data collection. Because the
second cohort begins one semester later than the first, adding an extra semester
of data collection on to the total. Next we answer the question, what is
the earliest grade level a student could begin participating in or receiving the
intervention? And the answer to that in our example is their first semester in
college because it’s a first-year seminar for students in their first and
second semesters and they begin in their first semester. Then the next question
asks, what is the latest grade level that students could begin participating in or
receiving the intervention? And in our example again the answer to that
question is the first semester in college, but there you can imagine
other interventions that may serve students in multiple grade levels,
beginning at different times, different grades. Okay, next we proceed through
questions six through nine and for question six, there are three
different parts. The first part asks, is baseline measured at the same grade
level for all students? So in our example, we answer yes and we
indicate the grade level in response to questions 6b which says, if you answered
yes for 6a, what grade level is baseline measured for all students? In our example
that’s pre-college. If we had answered no to question 6a and said that
baseline is not measured at the same grade level for all students,
then question 6b would have disappeared and we would respond to questions 6c,
which would have asked, what is the grade level when baseline is measured? And we
would have selected from a drop-down menu options, such as the start of the
first semester of the intervention or one semester before the intervention and
the like, but because we’ve answered yes, then
question 6c has disappeared from our from our options. Next we go on to
question 7, which asks, what is the maximum number of semesters that
students can participate in the intervention or comparison condition?
Which is two semesters in our example. Question 8 asks, what is the highest
grade level during which students can participate in the intervention or
comparison condition? Which is their second semester in college in our
example. We have a two semester, one year intervention for
students that they participate in during their first semester in college and
their second semester in college. And then the last question asks, how many
semesters post-intervention will you collect data?
And in our example, as we’ve stated that’s two semesters post-intervention.
Okay next we can review the automatically generated matrix
on the matrix tab. Once all the design questions have been
answered, the matrix tab will provide a customized diagram that shows all the
groups of students that can potentially be included in the sample for impact
analyses. We’re going to look at the two versions of the matrices produced from
our example, one at a time. First we’ll look at the matrix produced by the
general version of the tool and then we’ll look at the matrix produced by the
cluster RCT version. So this is the matrix produced from the general version
of the tool. The layout of the rows is the same in both versions of the
matrix. The top row, which is shaded beige, provides a letter to label each of the
columns that have information about students in the sample. This enables the
user to refer to specific cells by naming the column letter and the row
number, for example cell a1 or cell a2. Then there’s the cohort row, which
provides the cohort number. Each cohort is shaded a different color. The shaded
color may vary depending on the number of cohorts in your evaluation. In this
example, cohort 1 is shaded blue and cohort 2 is shaded purple. Next is the
baseline row, which provides information about when the baseline variable is
measured for students shown in the same column. The baseline cells are shaded
orange and in our example baseline is measured before college so the cells say
pre-college for each column and each cohort. Okay next are the numbered rows and
these rows correspond to the progression of time in semesters relative to the
start of the intervention. Each row represents a semester, beginning with the
first semester that the intervention is implemented during which data will be
collected. In this example as well as in the cluster RCT example that we’ll be
looking at there five rows. Row one is semester one, which is the fall
semester of the 2015-2016 school year in our example. Row 2 is semester two, the
spring semester of the same academic year, spring 2016.
Row three is semester three, row four is semester four, and row five is semester five. So fall 16, spring 17, and fall 2017. In the general version of the matrix, the
columns correspond to each grade level within a cohort. In this example there
are two cohorts, each with students from one grade level, first semester freshman. Therefore, there
are a total of two lettered columns. Column A shows the progression of
incoming freshmen in cohort one from their first semester in college through
their fourth, and column B shows the progression of incoming freshmen in
cohort two. The cells in the columns include information about two aspects of
time. Grade-level, the cells show students
grade level in college in semesters as students move from
their first semester to their second semester to their third semester through
their fourth semester in college. And the cells also include information about the
amount of exposure, the number of semesters of exposure to the
intervention which depends on the duration of the intervention. So one semester of exposure to the
intervention or comparison condition is shaded red and occurs in fall 2015 for
cohort one in our example, and spring 2016 for cohort two in our example. Two
semesters of exposure is shaded green. And then, in addition to information
about the amount of exposure cells may also include information about the
number of semesters since the end of the intervention if data collection
continues past the end of the intervention. So in the example, data are
collected one semester post intervention, which is written as 1S post referring
to one semester post intervention and two semesters post intervention is
written as 2S post, And note that as described above, the
third aspect of time captured by the matrix in addition to grade level and
amount of exposure or post-intervention follow-up, is the time relative to the
start of the intervention, which as we mentioned before is represented by the
rows in the diagram. Okay now here is a the matrix produced by the
cluster RCT version of the tool. Assuming that we answered the nine
questions in the design question tab in just the same way that we did for the
general version of the tool. Okay and again remember the example is
an intervention that begins in fall of the 2015-2016 school year with data
collected for a total of five semesters, from two cohorts of first semester
students, who participate in a two semester intervention, with outcomes
collected for two semesters post-intervention. In the cluster RCT
version of the matrix, the layout of the rows is the same as in the general
version. The top row is shaded beige and provides a letter to label the columns.
The second row provides the cohort number, the third row provides
information about when baseline is measured for students shown in the same
column. The next set of rows corresponds to the progression of time relative to
the start of the intervention, with each row representing a semester. Again there
are five rows, one for each semester of data collection across the two cohorts.
In the matrix for cluster RCTs, just as in the general version of the tool, the
lettered columns also correspond to each grade level within a cohort of students.
However for each grade level in the first cohort of students, the columns are
divided into three parts distinguishing students’ status as stayers, early joiners,
or late joiners. In this example, there are two cohorts of students.
Each with students from one grade level, first semester freshmen. The first cohort
of students, instead of having one column of students in their first semester in
college, there are three columns. Column A shows the progression of incoming
freshman and cohort one who joined the cluster well after random assignment,
these are the late joiners. Column B shows the progression of incoming
freshmen in cohort one who joined the cluster soon after random assignment,
these are the early joiners. And column C shows the progression of incoming
freshmen in cohort one who were present in clusters at
the time of random assignment and are labeled stayers. All subsequent cohorts of
students are late joiners, because the students enter the clusters well after
random assignment, at least one semester later. In this example the second cohort
of first semester freshmen appears in column D and the cell number D2 is
labeled late joiners. Apart from distinguishing students’ status of stayers,
early joiners, or late joiners, the layout of the cluster RCT matrix is the same as
the general version of the tool. In the cluster RCT version, the cells in each
column include the same information previously described for the general
version about grade level and about amount of exposure to the intervention
or post intervention follow-up. Next is step 4, to consult the matrix
explainer tab. You want to do that to be sure you understand the layout of the
sample matrix. The matrix explainer tab provides illustrations and
definitions for each part of the diagram produced in the matrix tab. For
orientation we show what the matrix explainer tab looks like on the general
version of the multi-cohort tool. There are explanations for three aspects of
the diagram that are provided as a reference for the user. The first two
statements and illustrations focus on the baseline row and the cohort row, and
those are the same in both versions of the tool. But, the third statement and
illustration, which is about the remaining rows, showing the progression
of time that differs on the two versions. So we’re going to look at the
third statement for both versions and they’ll be described in this section. So
on the next set of slides we’ll look more closely at each of these three
statements and illustrations. The first statement shown on the matrix
explainer tab again is the same for both versions of the tool. It describes and
illustrates the cohorts row in the matrix. For each cohort, cells
are shaded a different color. In the example shown on the matrix explainer
tab, there are three cohorts labeled one for cohort one and shaded blue, two for
cohort two and shaded white, and three for cohort three shaded purple. Note that
there are four columns for each cohort, and as described previously, each
column corresponds to a grade level within a cohort. Therefore, four columns
per cohort occur when there are four grade levels in each cohort. The cohort
row does not provide information about grade level. Grade level information
appears on a later row in the matrix and is described in the third
illustration on the matrix explainer tab. The second statement shown on the matrix explainer tab is also the same in both versions of the tool. It describes and
illustrates the baseline row of the matrix. In the matrix baseline cells are
all shaded orange and the values are the grade level when baseline is measured. In
the example shown, baseline is measured at the beginning of the first treatment
or comparison semester, therefore the grade level differs for
each column depending on students’ grade level at entry into the study: their
first semester in college, second semester in college, third or fourth
semester in college. In the general version of the tool, the third statement
on the matrix explainer tab describes and illustrates the remaining rows of
the matrix. Here I’m only able to show the first sentence that appears on the
tab just to keep the text on the slide on the screen or readable size, but I
will go over all the information about the rows that’s covered on the matrix
explainer tab. So all subsequent rows have a heading to indicate the school
year and the semester, fall or spring, that’s represented by the row. And as
described previously, cells in these rows provide information about students’ grade
level and amount of exposure to the condition or the post-intervention
follow-up. Okay grade level, which is semesters
in college, is written as text without any shading. For example first semester
in college is written as first semester, second semester, third semester, and so on. Semesters of exposure to the intervention or comparison condition are
shown as shaded cells. In this example one semester of exposure is written as
one and shaded red, two semesters of exposure, written as two, is shaded yellow,
and three semesters of exposure, written as three, is shaded green. If data
collection continues past the end of the intervention, the number of
post-intervention semesters is indicated. One semester post intervention is
written as 1S post without shading. In the cluster RCT version of the tool,
the third statement shown on the matrix explainer tab also describes and
illustrates the remaining rows of the matrix. Again all the text that appears
on the matrix explainer tab is not shown here for space reasons, but I’ll go over
all the information. The rows follow the same structure as the general version,
with row headings that indicate the school year and semester, fall or spring.
And cells in these rows provide information about grade level and amount
of intervention exposure or post- intervention follow-up. The difference
between the cluster RCT and the general version is that the cells in the cluster
RCT version of the tool also distinguished students as stayers, early
joiners, or late joiners based on when they joined the cluster relative to the
timing of random assignment. Okay once the sample diagram or matrix is generated, the next step is to
identify the specific groups of students from each relevant cell in the matrix
that will be included in the evaluation sample used to address each research
question for each impact that will be estimated.
So impacts could be estimated for students with the same amount of
exposure to the intervention or comparison condition regardless of their
grade level or the point in time relative to the start of the
intervention. Or, impacts could be estimated for students at the same grade
level regardless of the amount of exposure to the intervention or time
relative to the start of the intervention. Or, impacts could be
estimated for students at the same point in time relative to the start of the
intervention regardless of their amount of exposure to the intervention or their
grade level. In order to identify the cells in the
matrix that will be included in the evaluation sample, you can create a
version of the matrix that can be copied or edited. To do this you click the
button on the matrix tab that says generate editable version of the matrix and a new spreadsheet will be created with a copy of the matrix. The
new version of the matrix can be edited and it can be copied and pasted into
other documents such as an evaluation plan. And then you can highlight the
cells in the matrix representing the students to be included in the impact
analyses for each research question or you can simply reference the relevant
cell numbers using the column letters and row numbers. I want to point out that
the multi-cohort tool for post-secondary interventions generates diagrams in
which academic terms are represented as semesters. For post-secondary
interventions that involve quarters or trimesters or other terms, it will be
necessary for you to modify the matrix to accurately reflect the actual
academic terms in your evaluation. Okay we’re going to go over examples for
how to use the matrix to identify the student sample for three different
research questions, focused on amount of exposure grade level or time point
relative to the intervention. So for amount of exposure, we can consider the
research question, “what is the impact of the intervention on persistence and
credit accumulation for students after one semester of the intervention, one
semester of exposure?” For grade level, we might focus on a research question like,
“what is the impact of the intervention on students at the end of their
sophomore year or their fourth semester in college?” And for time since the start
of the intervention, we could consider the research question, “what is the impact
of the intervention on students after the intervention has been in colleges
for three years at the end of semester six?” These examples are presented for
both versions of the matrix. First, we’re going to go through these three examples
for the general version and then we’ll go through the same three examples for
the cluster RCT version. Before we go through the examples of how to identify
the samples based on the research questions, I’ll introduce the matrix
showing all the potential samples, which will be the basis of our example
research questions. So this is a matrix generated by the
general version of the multi-cohort tool. The matrix would be produced
for a student RCT, a QED, or regression discontinuity design, but not a cluster
RCT. For a cluster RCT, you would use the other version of the tool. And, as I
mentioned will look at examples using the cluster RCT after we look at
examples using the general version. In this matrix there are three cohorts of
students who participate in the intervention or comparison condition, for
up to two semesters in their first and second semester in college or who
participate in the business as usual condition. During that time, our
outcomes will be measured each semester from fall 2015 through spring 2016. We begin with example one, ‘what is the
impact of the intervention on persistence and credit accumulation for
students after one semester of exposure to the intervention?” For this research
question, the student sample would be comprised of students in cells
representing one semester of exposure regardless of their grade level and
regardless of the program semester, the time since the start of the intervention.
In the example matrix, outcomes are measured for students in six cells shown
here outlined in red. These students have the same amount of exposure to the
intervention. All students will have had one semester of exposure to their
assigned condition when outcomes are measured. The students in the sample will
be different grade levels when outcomes are measured. Some will be in their first
semester in college and others will be in their second semester in college, and
students will have outcomes measured at different times relative to the start of
the intervention. Some students will have outcomes measured in semester one fall
2015 in this example, some in semester two, and
some in semester three. Similarly, samples could be identified to
examine outcomes for students with two semesters of exposure, which are shown here shaded green, and data would come from these three
cells, three groups of students. Or, outcomes could be measured for students
at two semesters post-intervention. In these examples, students are the same
grade level, but outcomes are measured at different time points relative to the
start of the intervention. Next we’ll look at example two, “what is the impact
of the intervention on students at the end of their sophomore year, their fourth
semester in college?” For this research question, the student sample would be
comprised of all the cells in the matrix in which students are sophomores,
regardless of their amount of exposure or the amount of time since the start of
the intervention. In this example the the sample would
include students in the six cells that are outlined in red. Students in the
sample would all be the same grade level, they would all be in their fourth
semester in college. The students in the sample would have different amounts of
exposure to the intervention. Some students will have one semester of
exposure and others will have two semesters of exposure. And students in the sample will have
outcomes measured at different times relative to the start of the
intervention. For some students outcomes will be measured in
semester three, for some in semester four, for some in semester five, and for some
in semester six. Similarly, samples could be identified to
examine outcomes for students at the end of their junior year, at the end of their
sixth semester in college, as shown here. Next is example three with the research
question, “what is the impact of the intervention on students after the
intervention has been in colleges for three years, at the end of semester six?”
For this research question, the student sample would be comprised of all cells
in the matrix corresponding to semester six, regardless of student’s amount of
exposure to the intervention or their grade level. In the example matrix, the
sample would include students in five cells shown here outlined in red.
Outcomes would be measured at the same time relative to the start of the
intervention. All students in the sample in semester
six will have outcomes measured. Students in the sample would have
different amounts of exposure to the invention. Some students will have one
semester of exposure and others will have two semesters of exposure. And
outcomes will be measured at different post-intervention follow-up time points.
Some students will have outcomes measured four semesters post
intervention, some three semesters post intervention, and some at two semesters
post intervention. And students in the sample would be at different grade
levels. Some students would be in their sixth semester in college, some in
their fifth semester, and some in their fourth semester in college. Okay now we can go through these same
examples again, but this time using the cluster RCT version of the tool.
So again we can look at three different research questions. One focused
on impacts after one semester of exposure, one focused on impacts on
students at the end of their sophomore year, the end of their fourth semester in
college, and one focused on impacts of the intervention after it has been in
colleges for three years, at the end of semester six. But before we go through
the examples for the cluster RCT version, I want to introduce the matrix generated
by the cluster RCT version of the tool, which forms the basis for our example
research questions. So it’s the same intervention
and design as we described for the general version, but the matrix looks
different. In this cluster RCT, there’s one cohort of schools that’s randomly
assigned to the treatment or comparison condition, and within those schools there
are three cohorts of students who participate in the intervention for up
to two semesters in college, or who participate in the business-as-usual
comparison condition during that time. And outcomes will be measured for each
semester from fall 2015 through spring 2018. With a cluster RCT design again
it’s necessary to specify whether the sample will include students who were
present in clusters at the time of random assignment, stayers,or whether the
sample will also include students who joined clusters soon after random
assignment. In other words, whether this sample will include stayers and early
joiners, or whether the sample will include any students who are in clusters
at the time that outcomes are measured. In other words, that the sample will
include stayers, early joiners, and late joiners. So returning to example one for
the cluster RCT design, we look at the samples that would be used to address
the research question of, “what is the impact of the intervention on
persistence and credit accumulation for students after one semester of exposure
to the intervention?” For this question, for the cluster RCT again we need to
determine whether the sample will include joiners or not. If we were to
focus on the sample of students that have one semester of exposure regardless
of grade level or semester since the start of the intervention, and focus on
students who were present in schools at the time of random assignment, a sample
with only stayers, then outcomes would be measured for students in two cells shown
here outlined in red. Specifically, students in the sample
would have the same amount of exposure to the intervention. All students would
have had one semester of exposure and all students would be stayers, students
who were present in schools or whatever the clusters are, at the time of random
assignment. Students’ outcomes would be measured at the same time relative to
the start of the intervention for all students. Outcomes would be measured in
semester one, but for students at different grade levels the sample would
include some students in their first semester in college and others in their
second semester. Similarly, samples could be identified to
include early and late joiners as well as stayers when examining outcomes for
students with one semester of exposure. So you could expand to include joiners
and many more groups of students would be included, including those from cohorts
2 & 3. Or if you wanted to examine impacts on students with two semesters
of exposure and includes stayers, early joiners, and late joiners, then the
sample would include students in five cells, the five cells with a green bar
showing two semesters of exposure and these cells are shown outlined in pink. Next we’ll look at example 2, the
question about impacts on students at the end of their sophomore year or the
end of their fourth semester in college. The sample will include students at the
end of their sophomore year, fourth semester in college, regardless of their amount of
exposure to the intervention or the time since the start of the intervention. And
then if the impact analysis focuses on sophomores who were present in schools
at the time of random assignment and soon after, in other words both stayers
and early joiners, then outcomes would be measured for students shown in four
cells, those cells that are outlined in red. Specifically, students in the sample
would be the same grade level, all students would have completed their
fourth semester in college. The sample would include students
present in schools at the time of random assignment or soon after, stayers and
early joiners. Students would have had, in the sample, would have had different
amounts of exposure to the intervention. Some with one semester of exposure and
others with two semesters of exposure. For all, then for all students
outcomes will be measured two semesters post-intervention and outcomes will be
measured at different time relative to the start of the intervention. For some
students, outcomes will be measured in semester three and for other students in
semester four. Similarly, samples could be identified to
examine outcomes for students at the end of their junior year, their sixth
semester in college, including stayers and early joiners, which are shown here
outlined in pink. Last is example three for the cluster RCT design, addressing
the research question, “what is the impact of the intervention on students after
the intervention has been in colleges for three years?” Measuring
outcomes at the end of semester six for the cluster RCT design. Again we need to
just specify whether the sample will include stayers, early joiners, and or
late joiners. If the impact analysis focuses on all students in
semester six, regardless of the amount of exposure or grade level, and regardless
of when they entered schools, in other words a sample that includes stayers,
early joiners, and late joiners, then outcomes would be measured for students
in five cells shown here outlined in red. Specifically, outcomes would be measured
for students at the same time relative to the start of the intervention.
Outcomes would be measured at the end of semester six. Students in the sample
would include those who were present in schools at the… all students who were
present in schools at the end of semester six, regardless of when they
entered the schools, whether they were stayers, early joiners, or late joiners.
Outcomes would be measured for students at different grade levels. Some students
in the sample would have completed their fourth semester in college, others would
be in their fifth semester in college, and others would be in their sixth
semester in college. Outcomes would be measured at different time points post intervention. Outcomes would be measured two semesters post intervention for some
students, three semesters post intervention for some students, and four
semesters post intervention for some students. All
students would have had two semesters of exposure to the intervention. Okay and that’s all three
examples for both versions of the tool. So from this webinar you should know how
to select the correct version of the multi-cohort tool for post-secondary
education interventions for your evaluation. You should be able to use it
to generate a customized matrix for your evaluation design that allows you to
identify the samples to include in your impact analysis based on the focus of
the evaluation research questions. You may find it helpful to revisit the tool
and your customized matrix over the course of the evaluation, especially if
changes occur in the intervention or your evaluation plan. I want to point you
to some additional resources for the multi-cohort tool. There’s of course the
multi-cohort tool for post secondary interventions itself, the tool itself.
Both the general version and the cluster RCT version. And there’s also a user
guide. All of these things are available online on the First in the
World evaluation technical assistance website or the IES evaluation technical
assistance website. And finally if you’re conducting an evaluation that’s focused
on an intervention serving students in Kindergarten through grade 12, there is
an analogous tool, K-12 multi-cohort tool and user guide, which are forthcoming.
They’re not available yet. And finally the recording of this webinar is
also available online on the First in the World evaluation technical
assistance website and the IES evaluation technical assistance website.
Thank you.

Leave a Reply

Your email address will not be published. Required fields are marked *