Methodology & Demographics - The Center at WestEd

THE CENTER’S CA INSIGHTS 2017-18:
Methodology & Demographics

Since 2015, The Center for the Future of Teaching & Learning (CFTL) at WestEd has conducted a portfolio of research studies aimed at understanding how California schools and districts are implementing the California State Standards in mathematics, English language arts (ELA), and science.

Three complementary research strands make up our Insights into California State Standards Implementation project: interviews with district curriculum and instruction leaders; focus groups with educators in a variety of school- and district-based roles; and surveys of teachers and school leaders drawn from a representative sample of the state. Data from each of these three strands are collected and analyzed on a yearly basis, each capturing different dimensions of the implementation experience while also enabling triangulation of findings across data sources. This page describes the methodologies behind our CA Insights work and shares some descriptive statistics about each research strand sample. View demographic breakdowns of previous research cycles in our archive.

Strand 1

District Interviews

Each year, The Center identifies over 40 districts throughout the state for targeted phone interviews with district curriculum and instruction leaders. The 2017 data collection wave included interviews with 45 district leaders representing all 11 service regions of the state recognized by the California County Superintendents Education Services Association (CCSESA). Districts were chosen to reflect the overall composition of the state in terms of district urbanicity (urban, suburban, town, or rural), the racial/ethnic makeup of the student body, the proportion of students eligible for free and reduced (FRL) lunch, the proportion of English learners (ELs) served, and the average pupil-teacher ratio.

Phone interviews were conducted from October 2017 to March 2018. During the one-hour call, district leaders were asked about their capacity, constraints, and decision-making related to implementation approaches; funding; organizational structure; process, measurement, and impact; professional learning and development; and instructional materials. Inductive coding methods were used to convert qualitative interview data into discrete categories for quantitative analysis. The Center is developing and reporting on findings from these data throughout 2018.

Demographic Profile: District Interviews

Strand 2

Focus Groups

The CFTL’s focus groups are designed to understand how schools are supporting teachers in their implementation of the standards. During the first two years of CA Insights research, the Center spoke with teachers, school leaders, and instructional coaches from across the state. In our most recent cycle in 2017, we shifted our emphasis to principals and instructional coaches specifically, with the aim of gaining a more developed understanding of how schools employ a range of teacher support roles–including teachers on special assignment, teacher leaders, data coaches, specialists, and site leaders–in their standards implementation strategies.

In the fall of 2017, principal and instructional coach focus groups were conducted separately in four regions of California: Yuba City, Fresno, San Francisco, and Long Beach areas. Each group session lasted for two hours and consisted of six to 10 participants. Selection of participants was based on strict criteria: each group had no more than two participants from a single school district, each was evenly divided between participants from primary and secondary grade levels, and each featured wide variation in years of experience. This selection process resulted in 30 principals and 35 instructional coaches representing all K-12 grade bands and 43 districts.

The research protocol was carefully developed by speaking with educators and stakeholders from across the state, including representatives from county offices of education, education policy leaders, researchers, and technical assistance providers. Their feedback concerning what they sought to learn about the current state of standards implementation informed the kinds of questions and topic areas covered in the focus groups. The Center then analyzed the transcripts using categories derived from the protocol. Specific categories developed for coding included: implementation approach; instructional shifts; professional learning and development; data and assessment; and instructional materials.

Demographic Profile: 2017 Focus Group Composition

Focus Group Composition by District Locale

Focus Groups: Racial/Ethnic Composition of the Student Body

District Interview and Focus Group Comparison: English Learners and Free and Reduced Lunch Eligible

Strand 3

Teacher and School Leader Surveys

Since 2015, the Bill and Melinda Gates Foundation’s Measure to Learn and Improve (MLI) initiative has examined the implementation of college- and career-ready standards across multiple U.S. states through surveys of teachers and school leaders. WestEd serves as the California Data Partner for the MLI initiative. The Center’s third strand of CA Insights research comes from annual MLI data drawn from the RAND American Teacher Panel (ATP) Survey and the RAND American School Leader Panel (ASLP) Surveys.

About the Data Source: The RAND American Teacher Panel (ATP) & American School Leader Panel (ASLP) Surveys

Designed to survey the same educators at regular intervals over time, the RAND Corporation’s American Teacher Panel (ATP) and American School Leader Panel (ASLP) surveys launched in 2014 and have been administered multiple times yearly since. The ATP and ASLP surveys are nationally representative, with oversamples in four states (California, Louisiana, New Mexico, and New York) to ensure state-level representativeness in each. RAND currently uses single-stage, stratified random sampling to draw new teachers and principals at random for its panels, from lists contained in existing, comprehensive databases (stratifying on the basis of state, for oversampling). Each panelist is assigned a weight that accounts for sample design and probability of non-response (taking into account key variables like school size, grade span, poverty status, and experience), and linearization techniques are used to calculate margins of error. Panel members complete three or four web-based surveys every year, and panelists receive a small monetary incentive after completing each survey.

To create the ATP, RAND researchers first sampled 2,300 public schools from the 2010/11 and 2011/12 federal Common Core of Data (managed by the U.S. Department of Education’s National Center for Education Statistics), and stratified that sample on five variables: grade span (primary, middle, high and combined); school size; poverty status (proportion receiving free/reduced-price lunch); population density; and geographic region. Then, using probability-sampling methods, two regular full-time teachers were selected from each sampled school to join the ATP, and these teachers were stratified by experience level (novice and experienced) and subject area (three strata at the elementary level, and five strata at the secondary level). Individuals who change schools remain on the panel; new members are added periodically so the panel remains representative of teachers over time.

A total of 798 California teachers were included in the ATP sample for the May 2017 survey administration, and 479 responded, for a response rate of 60 percent. Weighting, which accounts for differential sampling and non-response, was used to produce results representative of the full state sample (yielding an approximate margin of error of ±4.5 percent). For the May 2017 ASLP survey, 386 of 1,024 California school leaders (38%) responded. For the October 2016 survey administration, 523 California teachers were included in the ATP sample and 281 responded (54%), yielding an approximate (average) margin of error of ±6.5 percent. Only 45 of the 130 sampled California school leaders (35%) responded to the parallel October 2016 ASLP survey.