Bruce Baker, a professor at Rutgers and one of the best educational statisticians in the country, wrote a fascinating post today about potential legal issues arising from the movement to base teacher evaluations on gains on student assessments. This post is based on that one, so please go read it first if you plan to continue ...
Being a fellow blogger in the CASTLE blogging network, he asked for my feedback, so I wrote this in reply:
Bruce,
You are exactly on point. Both are legitimate legal problems, with the disparate impact being more of the slam dunk in my opinion. The disparate impact numbers would be off the charts and states would have a very difficult time establishing that it is a neutral policy. You start tying that in with school finance stats and other characteristics like age of school buildings and the picture is going to get very dark, very fast (no pun intended). This would have to be a disparate impact case, though, not a disparate treatment case (http://bit.ly/GpQJp) and since it is disparate impact, a class would likely be formed -- i.e. someone would need to invest money on the front end of this case to organize it - thus, the NAACP or some kind of organization like that would probably get involved.
The Due Process argument would be a harder (and much less profile) case, but it could be brought individually ... so we might start seeing a whole lot more of these. Your identified problems in the statistics I think are great, but you are one of the best statistical minds in the education field. Your average joe-blow lawyer would have a really tough time making that case. And, as long as these cases stayed at the district court level, that case would have to be made over and over and over again by lawyers in each distinct community within each distinct state. If those cases rose to the level of the Circuit courts or the Supreme Court, that would save lawyers some work, but it would still be a costly case to put together and perhaps not worth it to the teachers. Anyway, I think you are right on with the legal analysis, but I think a lot of things in the US don't make statistical sense, but the legal system is just not competent enough to always tease that out.
Another legal problem this would create is that if teacher evals were 50 or 51 percent based on test score improvements ... it would make it even more difficult legally to get rid of bad teachers whose student test scores happened to go up. You can put a bad teacher in front of an AP class, and those kids are still going to excel on the test. If that bad teacher has a bad personality, treats parents badly, or any other negative qualitative component for which she would otherwise be dismissed or non-renewed, the test score based evaluation just gave that teacher a silver bullet in court. Probably like your law person there at Rutgers, I teach my principals to not give a reason to pre-tenure teachers when RIFing, because if you give a reason, then you have to defend it in court. These polices not only give a reason, but they give a reason that is largely outside of the principal's control. Even if it winds up that courts still think that 40% negative qualitative evaluation is enough to still RIF or dismiss a teacher, the number of lawsuits is likely to go up dramatically.
Generally, all this is what happens when you start forcing statistics in the legal system - which is not built for that at all. The legal system is a very qualitatively oriented system, making decisions mostly based on evidence obtained through interviews and the like. The jury, even, is a qualitative system that collectively makes a decision based on all the evidence presented. Statistics throw a wrench in all that because people react differently to numbers. They think numbers don't lie (although, of course, we know that they can and do). That's why generally, I don't love policies that seek to make decisions based solely on numbers - these kinds of things are the result.