Value Added Evaluation Data and FOIA (State Versions, That Is)
Justin Bathon on Thursday, October 21, 2010 at 11:23PM Bruce Baker and Andy Rotherham brought my attention today to the issue of the public release of school data used in value-added teacher assessments (which are becoming popular, spurred by Race to the Top- Scott and my previous posts on the subject). This stems from a report out of New York today that the New York City Schools are set to release this value added data on 12,000 teachers ... tomorrow.
First, the basics. Public schools are public ... meaning taxpayer owned. Generally, then, the taxpayer owners (meaning the general public) have access to all the information produced and stored by these entities. In a democracy such a concept would make sense, right? Well, except when we are planning to bomb another country or when we are surveilling a criminal or when ... well, you get the idea, there are lots of exceptions. One other important point before we begin is that there is a federal Freedom of Information Act, but for this analysis we are concerned with state laws, as schools are state entities and school personnel are state employees. Here is a nice starting point for your state to get the basics.
The exception at issue today is that of personnel information. They cover the gamut from very restrictive to very broad to non-existent. Let's start with Wyoming's quite broad protection against release of personnel records (Wyoming Statutes 16-4-201):
(iii) Personnel files except those files shall be available to the duly elected and appointed officials who supervise the work of the person in interest. Applications, performance ratings and scholastic achievement data shall be available only to the person in interest and to the duly elected and appointed officials who supervise his work.
Then there are some provisions in the middle. For instance, here's Delaware's relevant provision (Del. Code Ann. 29 - 10002):
(1) Any personnel, medical or pupil file, the disclosure of which would constitute an invasion of personal privacy, under this legislation or under any State or federal law as it relates to personal privacy; ...
Arkansas is another similar to Delaware. But, other states are more restrictive in their provisions. Take Georgia, for instance (50-18-72):
(5) Records that consist of confidential evaluations submitted to, or examinations prepared by, a governmental agency and prepared in connection with the appointment or hiring of a public officer or employee; and records consisting of material obtained in investigations related to the suspension, firing, or investigation of complaints against public officers or employees until ten days after the same has been presented to the agency or an officer for action or the investigation is otherwise concluded or terminated, provided that this paragraph shall not be interpreted to make such investigatory records privileged.
Other states, such as Illinois recently, considered adding such an exemption for public employees to their Open Records Acts. This story out of Illinois puts the number of states with some type of personnel evaluation exemption at around 35 - and without better data or time to investigate it myself, we'll go with that - meaning that perhaps 10-15 states may have no exemption at all for such records.
Thus, even without considering the issue of value added data sets, in some states there is not even an opportunity to argue that such evaluation data is exempt. Therefore, before any teacher (or more likely union) argues that such data cannot be released they need to be familiar with their state's law governing open record exemptions.
Now, turning to the issue of whether raw value-added assessment data is exempt from public record laws let's consider a few different options.
Option 1: raw data contains information capable of linking to particular students = FERPA protects against release. But, putting FERPA aside ...
Option 2: raw data does not contain link to identifiable teachers (or students) = Grant open records request as no exemptions are likely to apply.
Option 3: raw data does contain link to identifiable teacher = probably release. Obviously this looks like the New York situation and the one Bruce and Andy were highlighting. However, the data is not an evaluation, it is simply raw data. That it affects an employee's future evaluation criteria is probably not relevant if the open records request is only for raw data. Obviously this is going to be the case most dependent on the statutory language. For instance Wyoming's inclusion of "scholastic achievement data" seems a good bet to open the door for exemption even of the raw data. Delaware's "relates to personal privacy" is a crack, but a small one and one I don't think most courts would extend such a amorphous concept to this extent. Georgia, Illinois and other states, though, would have no legal avenue available to block release. At best, the teacher or union might be capable of delaying the release. Given the fast-paced news cycles, though, and the lack of persistence among researchers, delaying the release even a few weeks might be good enough.
Option 4: post-analysis, teacher-specific data leading to evaluation ratings = probably protect. This is where the easiest case is made to exempt the records, but still not in all states. If you are in a state with a personnel records protection provision, the evaluation analysis of the data is probably protected as I think most rational people consider job performance ratings to be personal in nature. This protection is not trivial, however, as working with the raw data is not a simple task and puts a much larger burden on journalists, researchers or other individuals requesting access. Obviously it is just much easier for them to publish the school's analysis and evaluation of the teacher. So, still not totally protected in this instance, but the easiest legal case to make as the statutes seem to imply, if not explicitly state, that such evaluations are protected.
Option 5: I think it is possible to visualize a hybrid between Option 3 & 4 - or perhaps several (help me out Bruce as you know the data better than I do). The analysis of the value-added raw data would likely involve several steps, each of which would get us closer to protection as the data goes from unaggregated to aggregated to compared against previous years to compared against similar classes to graphed to assigned a rating code to translated to an evaluation statement. Where along that line of analysis protection begins ... well, that's the kind of question that takes decades of litigation. Each specific instance will be different, meaning there could be minimally 50 and potentially thousands of different scenarios possible as individual districts begin rolling out value-added systems.
Okay, getting back on track then, this explains the New York story, for now anyway. New York has a mid-to-narrow exemption (see Section 87(2)) putting it somewhere between Delaware and Georgia (if I'm missing something, New York readers, let me know). Thus, as New York City's attorneys have concluded, the likely position is that the raw data should probably be released.
That's not the end of the story, though, for the union. Their next play, in the absence of a protective statute or regulation, is to attack the raw data as unreliable and thus (1) lacking a rational basis, (2) a violation of due process, (3) unconstitutionally vague, (4) discriminatory or some such effort. Remember, though, this is not an open records issue, it is attacking the facial validity of the entire value-added evaluation system (read the posts that Scott, Bruce, & myself wrote before). As Scott explained these efforts rarely win, but I kept the door slightly open if there is overwhelming evidence of statistical error. This is going to be an uphill battle, however, because the overriding public policy at play here dictates open government over privacy. That the rationale behind open government statutes is so clearly stated and privacy law is such a legal morass is not going to help this situation either.
Anyway, summarizing, I think the raw data is generally going to be made publicly open following FOIA requests. I think New York City is currently correct in their assessment that no exemption exists under New York's Freedom of Information Law. However, this is just my analysis after considering this issue for a single day and I want to caution against over reliance on my initial assumptions. A thorough analysis needs to be conducted of all 50 state policies, interpreting regulations, attorney general opinions, and previous case law. Further, data experts such as Bruce must assist the analysis with a complete understanding of each state's dataset and the possible links to both teachers and their evaluations within the datasets. Thus, there is still a lot of work left to be done.
This is a legal frontier (another one of those enabled by technology) that most legislatures would not have contemplated as possible in enacting their open records laws. Thus, it is a great topic for us to debate further to inform future policy actions on open records personnel evaluation exemptions.
Reader Comments (13)
Justin, interesting issue--and one that had not occurred to me (thus, I have to get up to speed on the records release issues before commenting about a good deal of this stuff).
As to the claims you mention in the third-to-last paragraph, are we still talking about privacy claims here? It seems not. It seems that we are talking about wrongful termination claims. Where you mention irrationality, discrimination, etc., it would seem that the unstated assumption is that some kind of employment decision is going to be made based on these released data. It may not be. Rather, this is more likely to be used as a public "shaming" mechanism, as it was in California recently.
If that's the case, then any claims would likely take the form of common-law tort claims for "public disclosure of private facts." These claims are hard to win, but they are the primary vehicle through which individuals protect their FERPA rights, and I see them as similarly applicable to state records release statutes.
Here are some data scenarios:
A. The district has individual student test score data that are linkable to individual teachers but the district doesn't use those data to generate any estimates of individual teacher "effectiveness," has not adopted any statistical method for doing so and therefore does not include any such estimates as part of personnel records. The individual students' identity can be masked but with matched ID over time and specific characteristics attached (race, low income status)
B. The district has individual student test score data that are linkable to individual teachers just as above, and the district does a) have an adopted statistical model/method for generating teacher value added "effectiveness" scores, but uses those estimates only for district level evaluation/analysis and not not for individual teacher evaluation.
C. The district has individual student test score data that are linkable to individual teachers as above, and a) the district has an adopted statistical method/model for generating teacher value-added "effectiveness" scores and has negotiated a contractual agreement with teachers (or is operating under a state policy framework) which requires inclusion of the "effectiveness" scores in the formal evaluation of the teacher.
Under option C above, sufficient technical documentation should be available such that "effectiveness" estimates could be checked/replicated/audited by an outside source. That is, while there should be materials that provide sufficiently understandable explanations such that teachers can understand their own evaluations and extent to which their "effectiveness" ratings are, or are not under their own control, there should also be a detailed explanation of the exact variables used in the model, the scaling of those variables, etc. and the specification of the regression equation that is used to estimate teacher effects. There should be sufficient detail to replicate district generated teacher effectiveness scores.
That aside, a few different scenarios arise.
1. The LA Times scenario, as I understand it, falls under the first conditions above. The data existed in raw form. The district was not using those data for "effectiveness" rating. The LAT got the data and handed them over to Richard Buddin of RAND. Buddin then estimated the most reasonable regression equation he could with the available data and, for that matter, produced a sufficiently detailed technical report - such that anyone accessing the same data could replicated his findings. I suspect that individual student names were masked, but the students were clearly matched to identifiable teachers, and student data included specific identifiers of race, poverty, etc. and participation in programs such as gifted programs (indicator on child that he/she labeled as gifted). Not sure what, if any, issues are raised by detailed descriptive information on child level data. In this case, the data requested by LAT and handed over to Buddin were not linked to teacher evaluation by the district itself, in any way, as I understand it.
2. As I understood the recent NYC media flap, the city itself was looking to report/release the value-added ratings and the city itself also intends to use those "value added" ratings for personnel evaluation. It sounded to me that Charleston, SC was proposing roughly the same. Each teacher would have a nifty little report card showing his her "relative" effectiveness rating compared to other teachers. This effectiveness rating is essentially a "category" labeling a teacher as "better than average" or "worse than average." These categories are essentially derived from more specific "estimates" which come from a statistical model, which generates a coefficient for each teacher's "effect" on the students who have passed through that teacher's classroom (these coefficients having substantial uncertainty and embedded bias which I have discussed previously... but that's not the point here). So, the effectiveness profile of the teacher is an aggregation of these "effects" into larger categories - but is nonetheless directly drawn from these effect estimates generate by the district itself for teacher evaluation purposes (even if subcontracted by the district to a statistician). I would expect that the specific estimate and the profile aggregation would be part of the teacher’s personnel record.
So, now that the city’s official release of effectiveness profiles is on hold, what if a local newspaper requested a) the raw student data linkable to teachers, with student names masked but with sufficient demographic detail on each student and with identifiable information on teachers, and b) the detailed technical documentation on the statistical model and specific variables used in that model? The newspaper could then contract a competent statistician to generate his/her own estimates of teacher effectiveness using the same data used by the district and the same method. These would not be “official” effectiveness estimates, nor could the media outlet claim them to be. But they would be a best attempt at a replication. Heck, it might be more fun if they used a slightly different model, because the ratings might end up substantially different from the district’s own estimates. Replicating or not, the districts own methods, and producing roughly the same or very different ratings for teachers, these estimates would still not be the official ones. Given the noise and variation in such estimates at the teacher level, it might actually be pretty hard to get estimates that correlate substantially with the district’s own estimates - and one would never know, because the district official effectiveness estimates for teachers would still be private.
I would assume under these circumstances, partly because the “official” personnel file estimates would remain unknown, and because it’s highly probable that the independent estimates produced by the media outlet - even if trying to replicate district estimates - might vary wildly from the district estimates - that the media outlet could get the data, estimate the model and report their results - their unofficial results.
Bruce, very helpful. A couple of questions:
1. Is getting the raw data publicly for scenario "c" really plausible? This level of disclosure, where it relates to an individual employee's personnel evalution, sounds like something you would need a lawsuit and discovery to get. That said, if a party were able to get the data and the evaluative interpretations of it, that party could certainly employ a person of your expertise to show how noisy it is, and as we discussed in the last thread on this issue, that might or might not be enough to show that the use of the method is unconstitutionally irrational or arbitrary.
2. In response to No. 2 above, I think that such analyses would be very useful publicly, but to build on what I say above, I seriously doubt that the school district would make the raw data (not to mention its own regression equation) available, knowing that smart scholars would be able to show how noisy it is pretty easily.
Thus, from the legal perspective, I am left with one question: if the data and conclusions are being used as reflected in option "c," but the media only gets the conclusions and not the raw data, then does the law allow a teacher to protect his or her reputation from unfair damage due to the publishing of a conclusion based on a noisy equation?
This is a very complicated question, involving both defamation law and the First Amendment. For example, is a public school teacher a "public official" or "public figure" for First Amendment purposes, such that the standard for proving defamation per se is increased? If so, then is the relevant statistical analysis illustrating the noisy nature of the conclusion enough to show falsehood for the purposes of a defamation claim? I think probably not in both instances, but I don't think this precise issue has ever come up.
If I am wrong about the availability of raw data, and these data are released, then there is a strong possibility that, even redacted, these data can be reverse-engineered to reveal the identities of at least some students, which may violate FERPA and give rise to student claims for invasion of privacy.
I suspect that those contemplating these releases have not sufficiently thought these issues out.
Regarding point #1. I'm suggesting that the media outlet is trying to obtain the full raw data set in order to generate their own teacher rankings and that the data in question are not packaged as teacher evaluation data. They are linkable to teachers and the media outlet could estimate their own teacher effectiveness model, but those estimates would not be "official" district estimates. They may not even resemble the official estimates - and we'd never know.
As a separate scenario, which I believe you are raising above, if a teacher was dismissed over his/her value-added score, I would certainly expect that teacher's lawyer to request access to all raw data and technical documentation so as to employ experts to show the arbitrariness of the model specification and sensitivity of results to data noise and modeling choices. A whole lot of expert witness work to be done there!
As to your point about concerns over releasing the data to researchers who might show the volatility of the data, I guess that depends on who does the reporting. In LA, the Times did the reporting and they substantially controlled the public spin on the reporting, holding to a party line argument that these are good precise estimates of actual effectiveness. No-one else has obtained the data to show otherwise, yet. I suspect NYC school district would be able to control the spin as well, getting the slickest looking teacher "report cards" out there and convincing the public that the information is reliable and valid. It would be hard for academic researchers to generate the counter-spin - and especially hard on a tight enough time line. The public would still argue for ratings and use them for their own convenience, the same way the public uses High School and College Rankings. Pick the one you like. It turns into a witch hunt pretty quickly. So your additional questions about defamation above are interesting.
Finally, I agree with your point that there would be sufficient detail (or should be) on student characteristics and classroom location, that the data could be reverse engineered to identify at least some students.
Thanks so much for the clarifying points, Bruce. What I'm thinking now is that, if a news outlet reports as "fact" an inherently noisy judgment of a teacher's professional "effectiveness," then one way to challenge the conclusion is to file a suit for defamation. This would be defamation of the teacher's professional reputation, so the teacher's damages would be presumed if the claim is otherwise good.
That brings up the questions whether (1) a public school teacher is a public official or public figure (which the news outlet will argue); (2) whether calling into question the methodology used or the validity of the judgment made based on statistical analysis is enought to show that the conclusion is factually "false" (falsity of the statement being an element of any defamation claim); and (3) whether any such conclusion is a purported statement of fact, rather than a statement of opinion (opinion being an elemental defense to a defamation claim). These are actually pretty interesting legal issues, and I do not know whether they have ever been considered in this way. At a bare minimum, though, discovery in any such suit ought to make the raw (and probably unredacted) data available.
Great points fellas. Some thoughts:
1. The FERPA issue is a great one - and one that perhaps I undervalued assuming the data could be sufficiently masked. But, your point necessitating classroom level detail of assessments combined with demographic data would give rise to a pretty hefty FERPA issue. In a class of 30 with 1 Asian student ... the redaction is useless unless you also redact the demographic data - which defeats the purpose. I think I am starting to lean as this as perhaps the union's best legal argument at the moment because the Open Records statutes almost always include a provision for an exception for all other state/federal law. If classroom level + demographic data is necessary ... I think there is a case to be made for an exception.
2. Without FERPA, though, I don't think Bruce that the information you provided about the data or the media's use of it changes all that much on the personnel evaluation exception. I still think those exceptions are not going to be all that useful if the request comes for the raw data.
3. The discovery issue on a defamation or wrongful termination claim is a good additional point. That would certainly put all of this to the test.
As much as I think this is a bad idea and agree with you Bruce that the data and analysis is not sufficient, my feeling is that something like this is inevitable. There is simply too much public data floating around for an aggressive organization to not pick it up and start ranking things. Such a ranking of teachers or a teacher report card would be a gold mine - as US News has proven on the College side. The basic picture for free, the detailed information for cash. It's horrible, but perhaps inevitable. Certainly without additional policy action on the part of state legislatures, the law is not going to be a substantial barrier to this kind of effort. There are some small issues to work around, but no major roadblocks.
Scott, how is UK's defamation claim working out against US News? Everyone at UK hates those ratings and thinks they are substantially flawed (potentially costing us millions), but proving defamation in a case like this would be horribly difficult. What is "false" about their aggregation and reporting of actual data? How they choose to do the analysis and which factors are included are opinions, not very capable of being proved true or false. So, even if these teachers are not public figures, I think they have very hard claims to make. If they are public figures (which I still have substantial doubts that they are) proving malice here is nearly impossible. Even recklessness would be difficult as these operations are sure to hire people with Ph.D.'s in statistics.
I don't think a defamation claim is frivolous by any means, but I don't see much of a winning path for a teacher against a US News type operation. The better defamation case would probably be against the school or state education department, as any flaws arising in the data amounting to falsity would probably arise there.
I totally agree that the raw data would easily be separable from district use of that data for personnel evaluation. Therefore, but for potential FERPA issues, the raw data could be made available - for good or evil. That was my original reason for starting this conversation - that I perceived Rotherham's reasoning of creating a shield by claiming official personnel record to be dreadfully oversimplified. Indeed, this is inevitable. What becomes most important is how the public perceives and uses this information, and how different parties present the information. As I always explain to my students, just because you can do something (that it's legally permissible), doesn't mean you should. My biggest concern in the short run is that the media presentation of this information and school district presentation of it is so one sided - arguing outright that these are the best possible, precise, accurate ratings of teachers. The defamation question above, along with the dismissal questions, might yield the first rounds of fun legal challenges. Few in the academic community (if any) are arguing that public disclosure of teacher VA ratings is a good idea, or that the statistics are "good enough" to be used this way.
What we have here is the tip of an iceberg colliding with a can of worms. That's a hefty mixed metaphor that I'll try to untangle.
The tip of the iceberg: The Federal Government is investigating big money for states to develop "data banks" on students and teachers that can be "mined" There is presently no consideration of
--what information that can be deposited in these digital archives,
--who can access the information
--how the information can be used
Maybe a huge hot potato would be a better metaphor, but that's the drift.
The can of worms. Standardized achievement tests and their statistical manipulation.
The history of achievement testing has cloaked in secrecy the construction of the tests and their reporting and interpretation. The cloak is protected by a phalanx of arcane statistical formulas that very few people understand, and those who do are far removed from day to day operations.
When one gets past the phalanx and peels off the cloak, what is left is psychological theory of the 1900-1930's era.
You can get a peek at what's going on in a paper on the Social Science Research Network: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1335923
Value-Added Measurement is akin to the Holy Roman Empire. What is "added" is spurious and it's not "measurement." The apparatus tries to "adjust" statistically what can only be adjusted in real time. It beats the alchemists quest to change lead into gold by attempting to change reality with a statistical formula.
The application of the statistics to teacher evaluation doesn't meet the mathematical assumptions involved, but that's an incidental formal/technical matter. It's the real-time variables that are the concern. I'll give two examples:
Prior student achievement. That sounds like a reasonable thing to adjust for. But what goes into the "achievement" equation is a number on an ungrounded statistical scale. "Math" and "Reading" in the real world are very different instructional matters. But the test scales are highly correlated. If the equation were to adjust for "Reading" in "Math" and vice versa there would be no difference between (allegedly) effective and ineffective teachers. There is more to the story, but that's the drift.
It is widely believed that "Poverty" is an important cause of student achievement so it seems reasonable to adjust for it. Because VAM involves "gain" teacher effectiveness is so gauged. But the "gains" obscure the fact that kids with low scores haven't been taught..
They're "gaining" individually and collectively, but in the real world, they haven't been taught to read and do math. Their current teacher isn't going to get the job done, no matter what the evaluation.
The "best practice" of testing and VAM amounts to cruel and inhumane treatment of both kids and teachers.
The good new is that the LA Times and the NYC Schools have inadvertently brought the matter of the iceberg and the worms to public attention. It's up to y'all to deal with.
Totally agree with Bruce's last point. It may be legal to release and publish, but it does not seem smart.
That said, to answer your question simply, Justin, you can't defame a university, but you can defame a person. The whole reason that I see a potential for defamation claims here is that these stories target the professional reputations of individual teachers. And if these teachers are not public officials or public figures, then there will be no need to prove actual malice (intent or recklessnes). The falsehood of the factual statement that "Teacher B caused smaller learning gains in her students than Teacher A" should be enough if it can be shown to be "false." However, as you point out, the statement may be an opinion, and even if it is not, showing that it is "false" will be extremely hard.
Nevertheless, a few suits may teach these districts and news media outlets a lesson that you don't mess around with people's professional reputations carelessly.
In their defense, can the district or media outlet simply claim that "they believed the estimates to be accurate representations of teacher effectiveness?"
One could certainly show that estimated from one year to the next, or by different methods or on different tests that the teacher was incorrectly labeled - or that the teacher was given a label that has only a modest if not very low probability of being an accurate representation of the teacher's actual effectiveness. What is the standard for "false" in such cases?
And, when the district translates the effectiveness coefficient into a category of effective or not, to what extent can the district officials or media outlet claim that their statistical representation is merely an "opinion?"
Those are the very issues that I'm struggling with, Bruce. As to the "reasonable reliance" argument, that would only work if the teachers are public figures or public officials because otherwise, there is no fault-based element to a defamation claim. As to the "falsity" and "opinion" arguments, I think the teachers will be more likely to lose, but I can't say it's a slam-dunk for the districts. How wrong does a test have to be before it is legally "false"? How much judgment has to be exercised before an inherently empirically-based statement that a teacher is more or less effective than his or her peers can be considered an "opinion"? If I were a school district or a news outlet, I would not want to litigate a flood of claims to figure out the answers to those questions.
I have had some further thoughts on this topic, and as it happens, Sherman Dorn has been thinking about it, as well. Here is a link to his post on the matter, which creatively suggests consideration of the "false light" tort as a means for the teachers to protect their reputations: http://shermandorn.com/wordpress/?p=2643#comment-120 . My comment to Sherman's post is reproduced below:
We have been conducting a discussion of this issue over at Edjurist (http://www.edjurist.com/blog/value-added-evaluation-data-and-foia-state-versions-that-is.html), but none of us have considered the "false light" angle--we have been focusing on other privacy torts, such as "public disclosure of private facts" (I understand that the NY teachers were not initially told that these scores would be published in the news media), and the non-privacy tort of defamation (specifically, "defamation per se," because this flawed set of scores impugns the professional reputations of the teachers involved).
Each of these approaches is problematic when the information we are dealing with can reasonably be characterized as a "government document." If so, the publication of such information is generally privileged, which would knock out defamation and probably public disclosure of private facts (unless state privacy laws or FERPA protect the information in question, thus leaving space for possible "public disclosure" claims against government officials for improperly releasing the information).
Any editorializing by the news outlets would constitute opinion, so it would not qualify for either tort, though editorializing that purports to put a factual gloss on the information might create a "false light" harm in some states. You are correct that this tort is highly state-specific (Florida's Supreme Court just recently abrogated it). My information is not current, but I understand that the tort is also not recognized in New York's courts. Also, the element of primary importance to the tort is that the information, as portrayed, must be "highly offensive to a reasonable person." The "effectiveness" scores themselves may meet that standard, but probably not. Portrayals of the scores as inscrutible judgments of relative teaching effectiveness might be more likely to meet the standard, but as I say above, I do not think the tort exists in New York.
Assuming that the papers engage in editorializing that purports to put a factual gloss on the scores, perhaps the harmed individuals should consider proceeding on a "single publication" theory in a state where (1) this theory is valid; (2) the relevant news outlets have subscribers and paper boxes; and (3) the "false light" tort also exists. If so, they would not only have to prove publication and offensiveness. but also that the papers acted with "actual malice" (knowledge that their portrayals create a false impression, or reckless disregard of the likelihood that they do). The truth of the underlying information is generally not a defense. Barring this (I think unlikely) scenario, the teachers likely will have to suffer through yet another indignity at the hands of those with no good answers to educational problems looking for scapegoats.