YES, THE SRA IS OUT!

It’s the best time of the year for any Oakland data nerd. It’s when data starts raining from the district and state!

I was thrilled to see that OUSD has released the Strategic Regional Analysis (SRA) for 2016-17. This second edition of the SRA includes many data sets on OUSD schools across the five regions of the city (Central, East, Northeast, Northwest, West). Kudos to OUSD for this progress towards more transparency and to the Data & Assessment team for their impressive work. I’m excited to see how this data informs OUSD decisions.

For those of you who remember last year’s SRA, there are three key differences in this year’s:

  • Interactive data visualization: Instead of a printable ginormous report with static graphs like last year’s, this year’s SRA presents information in forms of interactive dashboards, enabling each user to examine the data for answers to his/her own questions. You can focus in on one school, compare regions, or choose to look at trends for the district as whole.
  • Less guided experience: Without the structure of a report narrative, users may miss interesting patterns and important conclusions from the data, particularly users who do not have a data background or comfort in navigating data visualizations. While the Executive Summary does include a few key takeaways, I’d love to see more. Later in this blog I’ll offer a few of mine.
  • Deeper dive: This SRA dives deeper than before in some topics; for example, Teacher Retention expanded into Teacher Retention and Experience Levels.

Key takeaway #1: It’s a myth that Oakland enrollment is drastically declining! I’ve heard lots of people say that OUSD has been hemorrhaging students over the past few years. But actually, the number of students in district-run schools has held pretty steady over the past five years, enrollment staying within ±1% change each year. There has actually been a 11% growth in public school enrollment over the past five years, but charter schools are capturing the majority of that growth, including a 29% bump from 2014-2015 to 2015-2016

Snapshot from SRA Dashboard on enrollment. *including ACOE and OUSD charters

Question to Consider: if OUSD district enrollment has remained steady in past few years and per pupil revenue has been growing, how have we found ourselves in this current budget crisis? (We’ll explore this question in a new series on the OUSD budget.)

Key takeaway #2: We need more quality schools! (I feel like a broken record harping on and on about school quality, but don’t we all want more good schools for our students?!)  The number of quality schools (green or blue on the SPF tiering) drop as we move up grades. 10 of 54 Elementary schools and 3 of 22 Middle schools are blue or green. There are 2 “green tier” High schools – not a single top-tier district-run high school in Oakland. That translates to less than 1 in 5 students who have access to a quality district school in Oakland.

Snapshot from OUSD SRA dashboard on SPF, showing overall SPF tier for high schools only.

 

Key Takeaway #3: Huge variation in attrition rates by region. The rate at which OUSD students stay in public schools (retention) as they matriculate to middle and high school varies widely by region. Maybe this isn’t big news to folks – but I thought it was interesting that in the more affluent Northwest region, nearly one-third of OUSD 5th graders leave the Oakland public school system for 6th grade! Compare that to only 8% in the East region. The same happens again, where another one-third of OUSD 8th graders in the Northwest region leaves the system (compare 10% in East region).

Snapshot from SRA dashboard on Attrition and Transition for 5th to 6th grade.

What are your key takeaways? Send me an email with your insight and a screenshot of the relevant data visualization and I’ll add them to a future post as I continue to dive into the SRA. #datanerdsunite

More to come—stay tuned!

### My name is Carrie Chan, and I’m Educate78’s newest staff member. I joined the organization as an Analyst, and I LOVE data (feel free to call me a data geek). As a former OUSD student, I also care a lot about Oakland public schools. That is why I am so excited about this new blog series, “Crunched!” which will take a data-driven approach to important, relevant questions facing Oakland public schools. Please email me with ideas or requests.

Share this post!

The Old and The New: Comparing the Academic Performance Index and New California School Dashboard

Two weeks ago, the state unveiled publicly the first iteration of the new CA School Dashboard, providing student performance data and finally replacing the old Academic Performance Index (API), which was last issued for 2012-2013. Data geeks rejoice!

The Good, Bad, and Ugly

The new CA Dashboard is multi-measure and includes not only academic indicators (i.e. SBAC, grad rates) but also indicators of student climate (i.e. suspension rates). That’s useful for providing a more comprehensive picture of schools. Happily, the new CA system also considers growth, not just absolute performance (take note, Betsy DeVos!). Importantly, it also makes subgroup performance more visible – critical to understand if a school is helping to close the achievement gap for historically underserved populations.

Comparison of Academic Performance Index (API) and New CA Dashboard

These are all important improvements. But it’s not user-friendly. Unlike the old API, which boiled down to a single number plus two decile rankings, there is no single summary score and no way to easily compare against other schools in the dashboard. For each indicator, you need to look at the additional “Five by Five Placement Grid Reports” to understand why the school received the color coding it did. Some of the data currently in the dashboard is old while other data points are missing. And the Dashboard is only offered in English and is not sortable / printable / downloadable. Talk about jumping through hurdles for anyone (particularly families) who want to use this tool to figure out how a school is doing.

*Although the CDE website doesn’t allow comparisons, EdSource published an awesome tool to enable easy comparison of schools and districts on each indicator.

How are Oakland schools doing?

With the addition of all climate metrics plus growth, I wondered how Oakland schools are rated in the new system compared to our rankings under the old API system. I had to make some assumptions: I converted the API decile rankings into 5 colored tiers and took a straight average of all the metrics to come up with single summary color for each school in the new system.**

** API rankings to color conversion: Deciles 1-2 = red, 3-4 = orange, 5-6 = yellow, 7-8 = green, 9-10 = blue. New Dashboard indicators to summary color conversion: I converted the color for each indicator into a numerical value (red=1; blue=5), calculated a simple average across all the indicators available for each school, and translated that average back into the corresponding color. Note that many indicators (such as college readiness) were not available in this first roll-out, and school tiers might change with the inclusion of this data in the fall.

Some observations:

  • A different distribution: Under the old system, Oakland has a depressing number and proportion of schools that were in the lowest two deciles – a left-tilting ski slope. Under this new system, we seem to have fewer terrible schools, more mediocre and fair schools, and fewer excellent schools – more like a bell curve. This is not surprising since the cutpoints for the new system were designed to create a curve.
  • Majority still unacceptable: Even with the new system scores across multiple measures, almost 60% of Oakland schools are still in the bottom two tiers (i.e. “red” and “orange” schools) – not that different from the API-era rankings.
  • Excellence in short supply: In the new system, some schools that historically scored well on the API don’t do quite as well – due to lack of growth, poor subgroup performance, or lower climate scores.

The bottom line: we have a lot of work to do to create the school system that our kids deserve!

Next time: The Power of the 5x5 and Did schools change their colors?

###

My name is Carrie Chan, and I’m Educate78’s newest staff member. I joined the organization as an Analyst, and I LOVE data (feel free to call me a data geek). As a former OUSD student, I also care a lot about Oakland public schools. That is why I am so excited about this new blog series, “Crunched!” which will take a data-driven approach to important, relevant questions facing Oakland public schools. Please email me with ideas or requests.

Share this post!

“Keep Our Oakland Teachers!” – Teacher Retention

The data gurus at OUSD recently released a dashboard focused on teacher retention rates in OUSD. (Side note: I’ve found this announcement page from OUSD Data to be very helpful in keeping track of their newest data releases and news!) I’ve been thinking a lot about teacher retention ever since our Celebrate Oakland Teachers Night (plus all those news reports about California’s dire teacher shortage) – so of course I had to dig deeper!

Question 1: How high (or low) is teacher retention in Oakland public schools?

Answer: Three-year retention ranges from 40% to 70%, varying school and region.

Specifically, middle schools and the schools in the Elevation network (which are the schools that the district is making an intensive effort to improve) have the highest turnover. On average, a student in an OUSD middle school will watch two-thirds of the teachers leave by the time s/he completes 8th grade. Yikes! It raises an important question: How, as a city and a community, can we ensure that our students with the greatest needs get our city’s best educators?

Figure 1. Average teacher retention rate at sites by Network. Courtesy of OUSD.
Schools in the Northwest have higher teacher retention than schools in the East and West – not coincidentally the areas that have higher poverty and crime.
Figure 2. Average teacher retention rate at sites by SRA region. Courtesy of OUSD

A few other things to notice in this data:

  • One-year retention clearly doesn’t tell the whole picture: On an annual basis, teacher retention rates in some regions don’t look so bad (>80%). But, an average three-year site-based retention rate of 54%* district-wide is definitely not good – half of the teachers are different in three years!
  • Retention rates have declined slightly over the past 5 years: In the recent years (’11-12 to ’15-16), teacher retention rate in OUSD has dropped a little more each year from 83% to 79.5%. The drop in retention rates from 2008 to 2011 was almost certainly driven by recession-era reductions in available teaching positions.
Figure 3. Retention of teachers in OUSD from 2006-2016. Courtesy of OUSD

Eagle-eyed folks will notice that this data is limited to district-run schools. Unfortunately, teacher retention data for charter schools isn’t collected centrally or comprehensively. This is a gap I’d love to see addressed moving forward to provide a citywide look at retention!

Question 2: Are teacher retention rates and school quality correlated?

Teachers are the people directly working with our kids every day in the classroom. We often hear that teacher longevity directly impacts the quality of students’ educational experience. As a data nerd, of course I need to ask: What does the data say?

Answer: Yup!

There does seem to be a relationship, with higher teacher retention rates at schools with a higher School Performance Framework (SPF) score. While there is wide variation in Tier 2 and 3 schools, every single school in Tier 4 or 5 has an average retention rate of higher than 70%.

But, here is a big data nerd caution: even though there seems to be a correlation, we don’t know about causation! This may be a “chicken or egg” question: does high teacher turnover drive low student achievement and school culture? Or are working conditions and environment at the school accounting for the correlation between high turnover and high student need? Or, are both school quality and teacher retention symptoms of other causal factors?

A musing: You know what data I’d REALLY like know? Retention not schoolwide or region-wide or district-wide, but retention of our most impactful teachers – i.e. those teachers that are instructional wizards, amazing with family relationships, or really hold down a school culture. I wonder if the new Teacher Growth and Development System (TGDS) would provide data on this…

Questions 3: What can we do about this?

Answer: Glad you asked!

While we know about factors that increase teacher retention from a national perspective (see TNTP’s Greenhouse Schools report), we don’t have enough data locally to really understand the root causes of what’s causing our teachers to leave Oakland classrooms. (Of course I’m biased because I always think data is critical to getting to the answers!) To understand what affects our teachers’ engagement and satisfaction, Educate78’s Teacher Advisory Group is leading a citywide teacher retention survey. The results of this survey will enable us to connect local results with the national patterns that Gallup has seen: i.e. the more engaged a teacher is, the more likely they will stay in their current job.

PLEASE HELP US SHED LIGHT ON THIS TOPIC: ENCOURAGE ALL OAKLAND TEACHERS YOU KNOW TO TAKE THE SURVEY!

(PS: check out here what we did with this data last year! If you’re interested in learning more, keepan eye out for TeachOakland Grant page)

*This an average of the different regions. I recognized that averaging across the regions is not the best approach mathematically since the regions all differ in the number of teachers compared to the pure district-wide average, but the data is not downloadable for me to calculate quickly.

### My name is Carrie Chan, and I’m Educate78’s newest staff member. I joined the organization as an Analyst, and I LOVE data (feel free to call me a data geek). As a former OUSD student, I also care a lot about Oakland public schools. That is why I am so excited about this new blog series, “Crunched!” which will take a data-driven approach to important, relevant questions facing Oakland public schools. Please email me with ideas or requests.

Share this post!

OUSD Superintendents: Now and Then

Collage of some of the Oakland district leaders from the past 50 years (not comprehensive, includes photos where available).

With the departure of Oakland Unified Superintendent Antwan Wilson last week, the search is on!

I have a sense of déjà vu: we’ve been down this road quite a bit. As a data nerd, I started wondering about our previous superintendents. Who has led OUSD in the past 50 years? How many superintendents have we really had? How long did they serve? Are there any patterns or trends? I’m hoping some of the facts might help inform our current search.

NOTE: Oakland’s superintendents have been Board-hired, State-appointed (due to 2 state takeovers, both within the past 30 years), and Interim. I’ve focused my analysis on the first two categories.

FAST FACTS:

Of the 24 leaders who guided OUSD in the past 50 years, 10 were interim (42%) while the rest were “permanent” superintendents, either Board-hired or State-appointed.

  • In the past 50 years, 43% of our appointed Superintendents had previous OUSD experience. In the past 20 years, only 29% of Superintendents had previous OUSD experience. This downward trajectory is concerning if we think Oakland background or context matters. But this requires thoughtful, intentional grooming of future talent – or we’re forced to hire from the outside, as we’ve done. For example, there hasn’t been a Black interim superintendent in the last 20 years and all 4 past Black appointed superintendents were from outside OUSD.
  • In the past 50 years, not including Interims, 6 of 14 (43%) of our Superintendents had previous OUSD experience. But in the past 20 years, the percentage has dropped, with only 2 out of 7 (29%) with prior OUSD experience. If we think Oakland experience matters, we need to do a better job grooming, retaining and rewarding our future local leaders.
  • Average tenure of Oakland Superintendents (not including Interims) over the past 20 years is 2.4 years – almost exactly how long Wilson stayed! If we look farther back to the past 50 years, the average is a little higher at 3.4 years. The longest service in the past 20 years? Anthony (Tony) Smith at 4 years. For comparison, J.W. McClymonds served 24 years as Superintendent of OUSD, over a century ago, from 1889 to 1913. We know that continuity of leadership matters in public school education because it takes years to make meaningful changes – so we need to do a better job finding leaders who want to stay for the long-term, and we need to support them so that they do.

Some Interesting Observations:

PREVIOUS OAKLAND EXPERIENCE & LONGEVITY ARE NOT COROLLATED!

I’ve heard a lot of people express interest in hiring a local, someone who has prior Oakland experience (myself included). But, over the past 20 years, the leaders with Oakland experience and those with zero OUSD previous experience stayed the same amount of time on average (2.6 years for OUSD experience vs. 2.8 years for non-OUSD experience). So, hiring someone local does not ensure longevity (although local may beneficial in other ways).  Food for thought: what DOES ensure longevity, if not existing ties to district?

OUR LEADERSHIP HAS NOT REFLECTED STUDENT DEMOGRAPHIC TRENDS.

Over the past few decades, Oakland’s population has shifted and OUSD students are now approximately 45% Latino (25% Black, 13% Asian, 10% White). But only 2 of our last 11 Superintendents (including interims) in the past 20 years share the racial background of our largest student population.

THERE’S A GLASS CEILING HERE, TOO.

Women are underrepresented in leadership roles across industries and organizations, and unfortunately, OUSD is not an exception. In the past 20 years, only 2 out of 7 Superintendents were female – despite the fact that staff are disproportionately female.

THIS DECISION MATTERS.

43% of our “permanent” superintendents in the last 20 years have been state-appointed. Today, unlike all those other times, we have a voice in who leads our district. Let’s use it! Please tell your board representative what you think is important in this selection!

If you are like me and really just want to see the cold, hard data, see here for more in-depth analysis I did to reach all of the above.

*P.S.: Data I would love to add: how many past Superintendents share the background of our highest need students? I think this is a powerful leadership quality.

**P.S.S.: Special thanks to Jean Wing, Executive Director of Data & Assessment at OUSD, for providing data on past superintendents dating back to 1870, and Mara Benitez for her thoughtful insights.

### My name is Carrie Chan, and I’m Educate78’s newest staff member. I joined the organization as an Analyst, and I LOVE data (feel free to call me a data geek). As a former OUSD student, I also care a lot about Oakland public schools. That is why I am so excited about this new blog series, “Crunched!” which will take a data-driven approach to important, relevant questions facing Oakland public schools. Please email me with ideas or requests.

Share this post!

School Performance Framework Deep(er) Dive: Part 2 – Who’s improving…and who’s getting left behind?

Oakland Unified reported in its November Board meeting that its schools had an average 0.15 point improvement (recall that scores are from a 1-6 scale). 51 schools showed improved scores while 35 decreased in scores. Of course, we know that averages often mask interesting differences. Let’s look at the distribution of OUSD schools’ growth in SPF scores.

That’s quite a distribution; many schools are making substantial gains (0.25+, which translates to moving up a quarter of a tier) but many others have lost substantial ground. We wondered if the gains are evenly spread or concentrated in certain tiers (i.e. are the top schools also improving the most overall?).

We put together this visual representation so that we can dig in deeper.

In other words, many of the low-performing district schools (Tier 1) have improved over the course of last year! Tier 2 schools had very similar growth stats: 64% of the Tier 2 schools improved in the last year. It was heartening to see the equity-driven efforts at OUSD making a positive impact.

Tier 5 schools largely did not improve, but also didn’t drop enough to yield a change in tier. (i.e. like the difference between an A and A-.)

One big surprise: many of the schools whose scores dropped were doing OK (Tier 3). 56% of schools who received Tier 3 status in 2014-15 performed worse in 2015-16. Sad! It’s like the problem of an average student: he or she receives less attention when the teacher focuses time on catching up the students who are behind (Tier 1, 2), while the high-performing kids (Tier 4, 5) have outside support to excel even if they don’t get attention in class.

Food for thought: what is the strategy for the Tier 3 schools (i.e. “okay but not great yet” schools) in Oakland? How do we help them go from good to great, without losing momentum on improving the Tier 1 and 2 schools? With OUSD deep in budget and prioritization conversations, this decision of how best to improve schools for as many children as possible will be front and center.

In a future post, we’ll will dive deeper into which changes most affected gains and losses in schools’ SPF scores (i.e. SBAC results or suspension rates? Whole school changes or closing an achievement gap?), and how those might connect to OUSD initiatives and priorities.

What’s Next for SPF?

Oakland has 130 public schools. So far, the SPF is currently only published for the 77 district-operated traditional schools. That’s only 64% of the public schools in Oakland! We have been glad to hear that OUSD is collaborating with charter schools to develop a version of the SPF for charters so that data on ALL public schools are accessible and can be understood by everyone.

### My name is Carrie Chan, and I’m Educate78’s newest staff member. I joined the organization as an Analyst, and I LOVE data (feel free to call me a data geek). As a former OUSD student, I also care a lot about Oakland public schools. That is why I am so excited about this new blog series, “Crunched!” which will take a data-driven approach to important, relevant questions facing Oakland public schools. Please email me with ideas or requests.

School Performance Framework Deep(er) Dive: Part 1 – Overview

Data nerds in Oakland rejoiced last November when the Oakland Unified School District released School Performance Framework reports for 2015-16.  After a soft launch internal to the District in 2014-15, this was the first public release of the School Performance Framework (SPF) – a snapshot of how schools are doing. We now have an easier, visual, way to understand the current state and trajectory of OUSD schools, with a plan to add all other Oakland public schools (i.e. charters) this coming year.

Why does the SPF matter?

The SPF is a HUGE step to making Oakland school data more visible and understandable to the general public. Like the stars system on Yelp or report cards for students, it puts relevant information into the hands of people who need to know in an easily digestible and visual form.

SPF Tiers

What’s special about the SPF?

When thinking about school quality, many people tend to gravitate to a single measure: results on standardized tests. Alternatively, word of mouth, and therefore often socioeconomically defined networks of friends and acquaintances, can define reputation more than any objective set of measurements. Both of these extremes are too one-dimensional, or worse, misleading.

The new School Performance Framework from OUSD is a much more equitable, comprehensive, and developmental.

For example:

  • It looks at both whole school populations AND subgroups that are often left behind in classrooms, such as low-income or African-American students. This highlights achievement gaps between groups of students that might not be immediately evident.
  • It captures a number of different factors that contribute to school quality. The SPF includes metrics related to both academic (state testing, reading levels, etc.) and climate (suspensions, EL reclassification rates, etc.).  Hooray for a multi-measure approach! All the metrics are scored and weighted, resulting in an aggregate score from 1 to 6 (rounding down to get the tier).
  • It considers both absolute performance AND growth. Knowing how much a school has improved is critical for highlighting historically lower-performing schools that are now making huge strides for students.

The overall result? The following bar chart shows the tier by tier comparisons for the soft launch year and last year. We see some mild progress overall, with a 33% decrease in the lowest tier, and a 33% increase in schools in the middle or yellow Tier 3. Tiers 2, 4, and 5, all essentially remained steady. My upcoming posts will delve into the details behind the movements up and down the scale for individual schools, and then what is underlying that in terms of student subgroups and correlation with OUSD programs.

OUSD SPF Scores from 2014-15 to 2015-16

 

By the way, Oakland has been leading the way on this. In September, the CA State Board of Education adopted an evaluation rubric for schools to replace the old Academic Performance Index (API) that is also multi-measure, includes growth, and absolute performance and looks at subgroups…sound familiar?!

Next up: Who’s improving…and who’s getting left behind? (Looking at 2015-2016 SPF results)

### My name is Carrie Chan, and I’m Educate78’s newest staff member. I joined the organization as an Analyst, and I LOVE data (feel free to call me a data geek). As a former OUSD student, I also care a lot about Oakland public schools. That is why I am so excited about this new blog series, “Crunched!” which will take a data-driven approach to important, relevant questions facing Oakland public schools. Please email me with ideas or requests.