Teacher Performance Data Stirs Evaluation Debate
NEAL CONAN, host:
This is TALK OF THE NATION. Im Neal Conan in Washington.
Last week, the heated national conversation about rewarding good teachers and helping or removing bad ones flared after the Los Angeles Times published the names of 6,000 teachers in the L.A. Unified School District, along with data that showed how well their students performed on standardized tests.
Teachers unions protested angrily. One leader said the story was irresponsible and outrageous. Others, though, said it's about time parents had some way to objectively measure their kids' teachers.
Everybody agrees teachers' performance is crucial to student achievement. There is, however, no consensus on the best way to grade the person standing in front of the blackboard.
Parents, what do you want to know about your child's teacher? And if you're a teacher, how can we capture teacher performance? Our phone number, 800-989-8255. Email us, firstname.lastname@example.org. You can also join the conversation on our website. Thats at npr.org. Click on TALK OF THE NATION.
Later in the program, retired Colonel Gary Anderson was with us as the invasion of Iraq got underway. Today, he joins us on the Opinion Page as a new phase begins in Iraq.
But first, grading the teachers. Jason Felch is an investigative reporter for the Los Angeles Times, co-author with Jason Song of the Times' investigative report, "Grading the Teachers," and he joins us from Carpentaria in California. Thanks very much for interrupting your vacation to talk with us today.
Mr. JASON FELCH (Investigative Reporter, Los Angeles Times): Hi, Neal, happy to be here.
CONAN: And where did you get this information about these teachers?
Mr. FELCH: We asked the Los Angeles Unified School District for seven years of standardized test scores, from every child who takes the test in the district. That data is linked to their teachers, and using that data, we were able to, you know, do this analysis on teacher performance.
CONAN: And you used a method called value added?
Mr. FELCH: Right, value-added analysis is a statistical approach that's been around since the 1970s. In essence, what it does is it compares an individual child's performance relative to his own prior performance. And the difference between what the child was expected to do and his actual performance on the standardized tests is what's called the teacher effect. It's an attempt through kind of sophisticated statistical tools to isolate the impact that an individual teacher has on a student.
CONAN: There are lots of other ways, though, to measure how much a student is affected by their teacher.
Mr. FELCH: For sure, and nobody claims this measure should be used in isolation or alone. The benefit of this measure is that it is first of all, it's objective.
A lot of the evaluation tools currently used around the country involve a principal sitting in the back of a teacher's classroom for about 20 minutes once every few years and formulating an essentially subjective opinion of how that teacher's doing.
This is robust in that it looks at several students over several years under that teacher. And it's objective, it's not subject to whether the principal happens to like this teacher or not. But it also has its limitations. It only looks at standardized test scores in math and English. So it doesn't tell us about all the other important things that teachers do in their classroom that obviously help kids.
CONAN: And did you just get these numbers and this statistical analysis and then just rush it into print?
Mr. FELCH: No. We've been working on this for the past year. And part of that process has been testing the reliability of this measure. So we, in addition to doing the statistical analysis, spent months visiting the classrooms of more than 50 teachers all over the district, watching what they were doing, talking to parents, talking to principals, talking to students and comparing the outcomes of this statistical analysis with the kind of on-the-ground sense that other people have of how these teachers are performing.
What we found in general is that for principals who spend a lot of time in their teachers' classrooms, this confirms, this statistical analysis confirms something they knew or suspected about the teachers' performance.
For parents, however, parents have very limited access to information about individual teachers, and teachers make a really big difference for their students. So the idea of making this information public is to give parents a tool and given, it's a limited tool but a tool to know something about the performance of individual teachers in a school.
CONAN: And I wonder, William Sanders, one of the designers of the value-added system told our education correspondent Larry Abramson that value-added analysis can accurately single out star performers and ineffective teachers, but even the best process can't make meaningful distinctions about the great majority who are somewhere in the middle.
Mr. FELCH: That's definitely what we found in our analysis. Imagine in your mind a bell curve of effectiveness, and what you see is that the vast majority of teachers fall somewhere in the muddy middle. And making accurate distinctions about people in the middle is not possible.
These are statistical estimates based on a robust number of students, but this isn't magic, and these are estimates. So what this approach can reliably do, we feel, is identify teachers on the margins of that bell curve. So highly, highly effective teachers who make a really significant difference for their students and highly ineffective students whose students are falling behind year after year, this system is very good at identifying those teachers.
CONAN: And I wondered: Were teachers given any opportunity to say hey, wait a minute, this isn't fair because?
Mr. FELCH: Yeah, so obviously, as we do in any news story we write, if we're going to mention someone's name in an article, we give them an opportunity to comment and review and give their side of the story.
This presented a challenge for us when we were publishing the names of 6,000 teachers. And so, one of the things that we've done over the last year is built this rather complicated back-end database which allows teachers to log on, view their score before it's ever made public, and we can confirm their identity as the person whose score that is and then allow them to post a comment.
So when our database went live on Sunday, what you saw was hundreds and hundreds of teachers who had already had an opportunity to review their scores, to post a public comment about it, giving the kind of further contextual information that everybody agrees is important when considering these scores in context.
CONAN: How were they informed that their names were going to be in the paper?
Mr. FELCH: They first learned about it through our newspaper. We published our first story three Sundays ago. And in that story, we announced our intent to make this database public by the end of the month, in August.
So teachers had several weeks to log on to our website, view their own score and post a comment if they chose to.
CONAN: Let's get some callers in on the conversation. Jason Felch is with us, investigative reporter for the Los Angeles Times and co-author of the recent series "Grading the Teachers," 800-989-8255. Email email@example.com. We'll begin with Linda(ph), Linda calling us from Cincinnati.
LINDA (Caller): Hello.
CONAN: Hi, Linda.
LINDA: Hi. I'm a retired special ed teacher, and I hasten to say that my students did not have any kind of problems that were related to their actual ability to learn, nor were they physically disabled for the most part.
The main thing I want to say here is that I am not happy with the use of a business model to rate education because education is relational rather than business. And I think that if parents really want to have an effect on their children's education, and they're really interested, then their job is to get to know their child's teacher and come to that teacher with as the first line of issue. If you don't like what your the kid's teacher is doing, then tell that to the teacher first.
I had children all on individual education plans. So I'm quite familiar with the, you know, not having to use the math things. But testing was not good for these children because it blocked them into situations where they could not function. So something is going to have to be done about that. I really just don't feel that you can use a business model...
CONAN: So is any system to measure teachers appropriate?
LINDA: I think it's a part of the measure. You know, I think that is one thing to consider. And certainly, to some extent, you can weed out ineffective teachers that way. But I think it also leads more to teaching-to-the-test kind of behavior, and I don't think that's what anybody wants out of an education.
CONAN: All right, and Jason Felch - thanks very much for the call does this kind of report put too much emphasis on standardized tests?
Mr. FELCH: Well, we hear that concern a lot, and I certainly understand it. Nobody wants, certainly in the classroom, teachers don't want to be forced to really narrow the curriculum and only teach things on the test.
We hear from other teachers who say actually teaching to the test isn't a terrible thing because the test is actually based on what California has agreed is are the things that students at that grade level need to know. So if teaching to the test means teaching your kids multiplication, it's not terrible to teach your kids multiplication, and if you do so effectively, then your kids will do well on the test.
So I think the caller's point is a good one, though. This type of measure only picks up on a certain range of performance for teachers and teachers are very aware that theyre dealing with human beings and that there's a lot beyond just math and English skills that are important in the classroom. And this can't capture all of that.
So I think everybody in this debate about how to evaluate teachers would acknowledge and agree, including ourselves, this should not be the sole measure of a teacher. But it's an important measure, and when you look closely at the research that's been done, it's a far more tested measure than many of the other metrics that are out there to grade teachers.
So, principal evaluations in the classroom - there's very little evidence to suggest that principals are very reliable in their subjective assessments of teachers. Go ahead.
CONAN: I need to ask: Some people would say, look, some schools are in wealthier districts than others. How do you take into account poverty and other such factors?
Mr. FELCH: This is a really important benefit of this approach that I think is a little tricky to understand but important to grasp.
So the caller brought up special education students. Teachers have very legitimate concerns that the students they're assigned to, who - oftentimes they have no control over that, that they're going that their performance is going to be reflected by those students. So what the value-added approach does is control for differences among students by using - by judging students by their own prior performance.
So a special ed student would be, his growth would be measured by his own prior growth. So the same would apply to a poor student or to a student who has parents who are not involved in his education outside of school.
To the extent that these things remain constant, value added, as an approach, controls for these outside-of-school factors and is fairly robust at isolating a teacher's impact and only giving credit to teachers for the things that they can control.
This is in contrast to the way that we grade schools, which is on achievement basis, and parents...
CONAN: We're going to stick to teachers for today, Jason. But thanks very much for being with us. And again, we appreciate you're willing to duck out from your vacation for a little bit.
Mr. FELCH: Happy to be here. Thanks, Neal.
CONAN: Jason Felch, an investigative reporter for the Los Angeles Times, co-author with Jason Song of the Times' investigative report "Grading the Teachers." When we come back, we'll talk with Kate Walsh of the National Council On Teacher Quality. This is NPR News.
(Soundbite of music)
CONAN: This is TALK OF THE NATION from NPR News. Im Neal Conan in Washington.
Last night at the Primetime Emmy Awards, one of the big winners was "Temple Grandin," the HBO biopic based on the life of the autistic woman who rethought the way we handle livestock - took home five awards, including Outstanding Made for TV Movie.
We talked with Temple Grandin as the show came out about her struggles with autism and what it's like to be played on TV by Claire Danes.
Dr. TEMPLE GRANDIN (Animal Science): The thing that was amazing was how Claire totally changed into me. You know, if you look at her from a picture from last night at a party, you'd never think that, you know, that she could become like me. I mean, she's just amazing in how she totally turns into the character that she's acting.
CONAN: She talked about a lot more than the HBO film. You can listen to that conversation at our website. There's a link to it at npr.org/blogofthenation.
Right now, we're focused on a different sort of evaluation: How to grade schoolteachers. Los Angeles Times database with information on 6,000 teachers has been published. It details how well their students performed on standardized tests and reignited the long-lasting debate on the subject.
Parents, what do you want to know about your child's teacher? If you're a teacher, how do we capture teacher performance? 800-989-8255. Email us, firstname.lastname@example.org. You can join the conversation at that website, too, npr.org. Click on TALK OF THE NATION.
And joining us now is Kate Walsh, president of the National Council On Teacher Quality. She's been kind enough to join us here in Studio 3A. And Kate, I have to ask you, this is currently about Los Angeles, but this is a nationwide conversation.
Ms. KATE WALSH (President, National Council On Teacher Quality): That's right. There are really two issues here. One is whether or not it is appropriate to use this value-added data to evaluate teachers. And then the second one is do parents have a right to have access to that data?
CONAN: Do they?
Ms. WALSH: I don't think so. That's my own view. I think it's problematic when parents are brought into what is essentially a process that the principal engages in each summer to decide where children will be assigned and into whose classroom.
And parents have a great deal of self-interest, their child, and are not able necessarily to stand back from those - from that child and decide what is best for a classroom, a school, a teacher.
So I think it's essentially akin to sharing confidential personnel information on a teacher, and I think that there's no other profession that you can point to where such data is made public.
CONAN: There is the Los Angeles school system, Unified School District, is one of the second largest in the nation - and is notorious for its inability to lose ineffective teachers.
There have been cases that have been strung out for years and years and years. Would not this kind of an evaluation make that process a little easier?
Ms. WALSH: Well, I'm a great proponent of using value-added data in evaluation. So, and I think that one of the reasons evaluations does not work well in the United States or elsewhere is because we don't insert any objective data.
So principal is still sorry for bad teachers, and let them pass through without a bad evaluation when in fact we're not thinking about children first.
So just want to, very clearly, separate the issue between using that data, and I think it's a very appropriate tool for principals to use, and whether parents should actually see that data.
CONAN: Shouldn't a parent this is these are public employees, paid for with public money, evaluated by public money. Isn't this public information?
Ms. WALSH: If you put it that way, it sounds like it should be, but I would argue that when it comes down to Mrs. Jones deciding whether her little darling Suzie ought to be assigned to one classroom or another, she's going to very much advocate that her daughter Suzie be placed with the stronger teacher. When, in fact, the principal needs to exercise a little professional judgment that maybe little Suzie could do better in another class for other reasons, other than the value-added test scores.
CONAN: Shouldn't the parent have access to why that kid is in that particular class?
Ms. WALSH: Well, if you're advocating that a principal have 400 different conversations with parents each summer to explain and defend these decisions.
But the real problem comes down to those parents who don't advocate for their kids. So I, as a middle-class, affluent parent, will very much advocate to have my children placed with the best teachers.
But guess who does not advocate? The parent who feels disenfranchised. So if I'm out there advocating, and the principal bends to the pressure I'm putting on her, but, you know, the children who don't have anyone advocating for them are the ones who stuck with the weakest teachers. And that, in fact, is the system we have now in the country, anyway.
CONAN: Let's go to the calls. Sienna(ph) is with us from Boston.
SIENNA (Caller): Hi.
CONAN: Hi, Sienna, we're on the air. Go ahead, please.
SIENNA: Okay, thank you. I just wanted to share a story because I'm actually a student in Boston, Massachusetts. And I had an experience where, two years ago. I was in an English class, and I did really well, and then the next year, I ended up being stuck with a teacher who I did not work well with, and I ended up getting really bad grades.
And I think that, in a way, it was definitely a reflection of the way the teacher was teaching, and after that, me and my mother did try to, like, talk to the teacher about it, and nothing changed because of the way the school system is set up.
Like, if teachers are there for a really long time, it's hard to actually make a change and fix that sort of situation. So I just wanted to share that.
CONAN: Okay, and Sienna, you can understand that if it was just you who had a problem, well, then it's you and that teacher, and that's an unusual situation. If it's everybody in the class, that's another situation.
SIENNA: Oh, no, no, it was everybody in the classroom.
CONAN: Okay, all right. Well, Sienna, are you still in school?
SIENNA: Yes, I am.
CONAN: And I guess school starts this week?
SIENNA: Yeah, it does. Actually, no, next week. It's going to start next week.
CONAN: Next week. You're lucky. Most kids are back this week - or maybe even last week. Sienna, good luck in school this year.
SIENNA: Thank you, bye.
CONAN: Bye-bye. Joining us now with a little perspective from the classroom is Doug Lemov managing director of Uncommon Schools, a network of 16 college prep charter schools in the Northeast, author of "Teach Like A Champion: 49 Techniques That Put Students On The Path To College." And he joins us from member station WAMC in Albany. And Doug(ph), nice to have you back on the program.
Mr. DOUG LEMOV (Author, "Teach Like A Champion: 49 Techniques That Put Students On The Path To College"): Nice to talk to you, Neal.
CONAN: And as part of your research, you evaluated a lot of teachers. Are there objective measure to evaluate teachers?
Mr. LEMOV: Well, I think data has to be a part of it. The report from the Times pointed out, it's the only truly objective measure. But I think one of the fascinating things that he found in the article, that we found in observing our own teachers, is that brilliance comes, and talent comes, in so many stripes and varieties.
I think there's a point in the article where he describes how different the top-performing teachers were, but they all had their own styles, and some were introverts, and some were extroverts; and some stuck very tight to the curriculum, and some freelanced.
And I think it underscores for the me the power of objective data, which is it sets people free to be accountable for results and not inputs and not someone sitting there saying you have to do it this way, you have to do it that way but saying you find the right way to the goal line, and we'll reward you for that, and ideally we'll even learn from what you do that gets you there. And, you know, that's a gift to everyone in the profession.
CONAN: And do you agree with Kate Walsh that this is information that should not be provided to parents?
Mr. LEMOV: Well, Kate probably knows better than I. I'm an operator of schools. I think that the way that we think about this at Uncommon Schools is three things about teacher evaluation. It serves three purposes.
The first is to make teachers better. You know, we have teachers who are hungry to get better. They want to understand when they're doing well so they can do more of it, and that's the first purpose.
The second thing is to identify success so we can reward it. And the third reason is accountability, and I think that's, you know, one of the things that our principals have going for them in our organization, is that they have real flexibility to manage people and to reward talented people and to coach, firmly, people who are struggling.
And I think when you have labor dynamics that make it very, very difficult, when you have to build a case for years and years to tell someone that they're in the wrong job, or they're in the wrong school - that they don't match with the school's philosophy, you force a set of evaluation systems that really are only about dismissal. And I think that's really sad.
And, you know, I think one of the things that I'm proudest of about our organization, Uncommon Schools, is that our evaluation system is really built around rewarding people and identifying talent and shining a light on it.
CONAN: Do you have any indication that the Los Angeles school system, or any public school system, to your knowledge, uses data like value-added evaluations to, say - to bring teachers in and say you need more work, and here's how we're going to help you and get better?
Mr. LEMOV: Well, you know, I don't know enough about what a school system the size of Los Angeles and other schools systems do. But I do know that, I know that there are schools and principals, because some of them are within our organization, that use all sorts of data.
And I think that Jason, the reporter, made a couple of good points about, you know, data is one very important piece of the puzzle, but ultimately, student achievement is a team sports, and so there are team goals and an organization, all of us pulling together.
So there are some teachers I know, some of our best teachers at Uncommon Schools, choose to mentor other teachers. It's possible that that could have a slightly negative effect on their value-added scores. So that's something I want my principals to be able to take into consideration.
CONAN: Kate Walsh, one of the theories behind this series was, in fact, this information was being locked in a drawer somewhere and ignored.
Ms. WALSH: That's absolutely right in Los Angeles. And there's a great amount of pressure from teachers groups, including teachers unions, to not use the data. In fact, I heard that the only reason that L.A. was able to publish this data and get use of it, was because it had never actually used it for making any personnel decisions. If it had used it, the L.A. Times would have been prohibited from accessing the data because it would have been privileged and confidential. So the school system there is in a really tough spot.
But it's not - there are plenty of school districts which do use this information. Not all of them use this information very well. There's a lot of ways to misuse this data. For example, you really need to have three years of data on a teacher before it's reliable. There's a lot of danger in overinterpreting what the data say because you're not just picking up - I know Jason mentioned that we are measuring the teacher effect from the value-added measures. But actually, you're measuring the classroom effect, and there's a lot of noise on that classroom. And I don't mean noise, literally, though that could be a problem, too, but, say, you know, a lot of the kids had a cold the week that the standardized test was administered.
CONAN: But shouldn't that even out over a period of three, four, five, six years?
Ms. WALSH: That's why you definitely need to have several years of data to overcome the noise, and the L.A. Times had that...
Ms. WALSH: ...so they did use it responsibly.
CONAN: Here's an email we have from Kenny(ph). As a current college student, I use websites, like RateMyProfessor, to see how other students have rated professors when signing up for class. I wish there was something like that along those lines for primary and secondary schools as well. Obviously, students are biased and teacher ratings' grades should not be based solely on student grades, but students' thoughts about teachers should be taken into account.
Obviously, that's much more prevalent in college, I think, Kate, than in high school and in grade school.
Ms. WALSH: But I think - I don't know if it was a male or a female writing that email, but I think it was a very good point. And we actually advocate for students, at least secondary students, to be able to complete an evaluation of a teacher, and whether or not that gets rolled into the official personnel file, that's another matter.
But I know in Boston, they have recently passed a new provision that allows student feedback to be considered. And the Gates Foundation, which is looking at this issue very carefully, is also looking in to see how much student feedback correlates with results. So I think that will be very useful, and I think students really do have teachers' value pegged.
CONAN: Kate Walsh is president of the National Council on Teacher Quality, also with us, Doug Lemov, managing director of Uncommon Schools and author of "Teach Like a Champion." You're listening to TALK OF THE NATION from NPR News.
And let's go next to Francesco(ph), Francesco with us from Baltimore.
FRANCESCO (Caller): Hi, good morning - good afternoon. My issue is about the data not being available to citizens. I mean, ultimately, we fund the public schools through our property taxes, which are local. That's the case in, I imagine, almost everywhere across the United States. So we should be able to hold our teachers accountable. I don't have kids in schools, but I'd like to be able to see how my funds are being used at the public school level.
And I think giving it to teachers, it would protect them from subjective principals. I mean, especially hearing that the use of tenure is done with teachers, I think we're definitely entitled to see how their performance stacks up.
CONAN: Objective data. And I think just to back that up, Doug Lemov, I think it's the secretary of Education, Arne Duncan, who said that, in fact, there are any number of school districts where 99 percent of the teachers get satisfactory to better than satisfactory ratings.
Mr. LEMOV: Which is the same as not scoring them at all...
Mr. LEMOV: ...which gives them no information. I also think, you know, the thing that Jason pointed out in his article that I thought was actually the most valuable thing from a policy standpoint was the opportunity to shine a light on talent, which is a thing that's empathically not happening.
And if you read the words of the great teachers in L.A. who were interviewed for the article, you hear somewhere between a bristle and sadness about never being recognized and never being honored for their work.
You know, one of the teachers is Zenaida Tan, who's in the lead of the article, says, you know, in my write-up, I was scored proficient and I was noted for being late to pick up my kids at recess three times. And here's a teacher who has fundamentally changed the lives of so many kids in her classroom.
So, you know, the opportunity to use evaluation to shine a light on talent is generally missed, I think, across the country. Just imagining what would happen if you took the dataset that the L.A. Times published and kept the bottom three quarters of it anonymous and just published the names of the top quarter of teachers in the country - I just know that teachers are people who want to be successful. They want to change lives. That's why they do the work. Imagine how powerful the nonfinancial compensation of being one of the teachers who is indentified as an incredibly high performer would be. Just that would cause people to strive to reach higher standards.
CONAN: Francesco, thanks very much for the call. Here's an email with an opposite point of view from Dorothy(ph) in Winston Salem. Parents are not objective about their kids. Parents will probably misuse this data. The school administrators have to make assignments of students to teachers without the interference of pushy parents. Administrators shouldn't have to justify every decision. Students will always encounter a variety of teachers and should learn to get the best out of each one. Are we going to hover over employers to make sure our kids only get assigned to a star supervisor? Aren't we carrying this individualized attention and assignment too far? Best for parents to teach their children how to excel even if they're not assigned to the top teachers.
And, Kate Walsh, I think you would be in agreement with much of that.
Ms. WALSH: That very much echoes my sentiments.
CONAN: Let's see if we can get one more caller in, and let's go to Carol(ph), Carol is with us from Santa Rosa, California.
CAROL (Caller): Hello. I taught for 30 years - upper grades, high school in San Francisco in the '60s and then 20 years in Vallejo, where I taught grades two through six. And I have noticed that administrators, when I went to get master's in curriculum, administrators don't have a lot of years of experience of teaching in the classroom, especially teaching under the guidelines that exist now.
CONAN: I don't mean to hurry you, Carol, but we just have a few seconds left.
CAROL: Okay. No Child Left Behind, the consequences at my inner-city school was science, social studies, P.E., art and music were removed from the curriculum, and that's where I had been making critical thinking, writing and reading skills. It was reduced to rote lectures by the teacher where you had a classroom where a third of the kids didn't speak English.
CONAN: Those standardized tests. Boo, as I hear that. Carol, thanks very much for the call. And I - I'm sorry to summarize your thoughts in just that one word, but I do - I am afraid we're out of time.
Our thanks to our guests Kate Walsh of the National Council on Teacher Quality. I wonder if you have an enunciation component to that.
(Soundbite of laughter)
CONAN: And Doug Lemov is managing director of Uncommon Schools and teacher and author of "Teach Like a Champion: 49 Techniques that Put Students on the Path to College." Our thanks to them both.
Coming up, we'll talk about Iraq with retired Colonel Gary Anderson on the Opinion Page. Stay with us. This is TALK OF THE NATION, from NPR News.
NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.