I notice that the SF Weekly
article which misleadingly portrayed two "small schools" as achieving superior performance on standardized tests, relied almost exclusively on the "similar schools" rankings of those schools for its "proof" of achievement. The similar schools rankings were
removed from the California Department of Education (CDE) website in early April, and are being recalculated, due to the discovery that one criterion had been inadvertently left out. However, even if the recalculated rankings prove flattering to the two small schools in question, it was surprising to see anyone base a claim of "success" on the similar schools ranking alone. Most educators take these numbers with a grain of salt, not only because the measure has proved to be so
flawed in the past (and this year’s snafu only reinforces that image), but also because of questions surrounding how "similar" the schools in any "similar schools" cohort really are to each other.
Pat McCabe, the CDE’s director of policy and evaluation, puts it
this way:
"There's a lot of disgruntlement around the similar schools rank," he said. "A lot of it comes from people thinking that schools (on their lists of 100) ought to look like them."
McCabe said schools are grouped together because of the amount of challenges they face, even if the types of challenges are different.
To understand how schools which are grouped together as "similar" can actually be quite different from each other, it is necessary to understand how they came to be considered "similar" in the first place. Each school is assigned a School Characteristics Index (SCI) number, based on many
factors, including student mobility, socioeconomic level, participation in various programs like gifted and talented (GATE) and migrant education, number of students with disabilities, race of students, and teacher credentials, among others. Each school is then grouped with the 50 schools whose SCI number is just above, and the 50 schools whose SCI number is just below, their own. The schools in this group are then resorted, based on each school’s API number, and are divided into deciles, with a ranking of 1 being the lowest. These decile rankings represent the "similar schools" ranking.
The "similar schools" ranking is at best a misnomer. As the CDE website
explains, "Each school is unique; therefore, it is impossible to find similar schools that match in every way. In order to form large enough groups of similar schools for meaningful ranks, the procedure used for each SCI allows for some differences between schools."
An observer might believe that schools which "look" the same would be considered "similar", but this is
not always the case.
Another school in the school district has similar students and almost exactly the same API score but a different "similar schools" rank. How can that be?
The SCIs are calculated using many demographic characteristics. Even if schools appear quite similar in some characteristics, they may differ with respect to others. Small differences in two school’s demographic characteristics can result in different SCIs and, therefore, in different groups of similar schools. If one school’s comparison group has a different range of API scores than the other school’s comparison group, the two schools’ ranks may differ.
Clearly, "similar schools" is a measure of something, I’m just not sure of what. Despite all of the CDE’s attempts to include more and more factors into the creation of the SCI, I still feel like they have overlooked a major factor in assessing how "similar" the challenges facing various schools really are. I think of it as "the chaos factor."
Chaos poses a challenge in the lives of so many children that I think it should be acknowledged as being as significant as race or mobility or family income level. Take, for example, two students who appear to be "similar" based on the factors currently used by the CDE. Both of them are the same race; both are considered to be of low socioeconomic status. Both speak English as their first language; neither one is disabled nor a participant in GATE or migrant education. Both come from families with similar parent education levels. However, one child lives with mom, dad, and younger sibling (or maybe with mom, grandma, and the sibling). One adult or the other picks the kids up at their free afterschool care program every evening by 5:45. The family eats dinner together every evening; the TV is turned off during homework time; the kids are in bed by 9:30. An adult attends at least one parent/teacher conference per year. The family has been in their current apartment for 15 years.
Now compare this to another child. This child is currently living with an auntie, but last year he lived 3 blocks away with grandma. Before that, he was with mom, but she is no longer able to care for him or his younger sibling. During the time he was with mom, the family lived in 3 different places in 4 years, but the child has remained at the same elementary school throughout. In the current home, the auntie provides dinner at night, sometime between 7 and 9 pm, depending on what time she gets home from work, and whether she had to stop at the store to buy food. The TV blares all the time; the kids set their own bedtime, as auntie collapses exhausted into her own bed right after dinner. No one has ever attended a parent/teacher conference. The report card comes in the mail, but it is never opened, and no one ever checks to see if homework is done.
By the CDE measures, these kids are "similar", but which one do you think has a better shot at success in school?
And lest anyone think that the chaos factor is limited to family stability, consider any child, from any socioeconomic group, who struggles with a learning disability. For the child who sees letters literally dancing across the page, or who looks at "23" but thinks "32", the internal landscape can be chaotic all the time, regardless of what his home life looks like. These kids too should be assigned a higher chaos score than the kid who lives next door, has an identical family life, but has no learning disability. Although this year, the CDE added "students with disabilities" to the list of characteristics to be used in computing the SCI, any parent with a child in special ed can tell you that it is a Herculean feat to get a child identified for special ed in many school districts, and that often children who clearly face challenges in this area are never officially identified by their school district at all.
Other students who would qualify for high "chaos" scores include those who have had a prolonged illness affecting their school attendance; those with a family member recently deceased; those with a family member with compromised health (for example, mother recently diagnosed with breast cancer, or sibling with life-threatening asthma); those who live in neighborhoods with high levels of gang activity, drug abuse, and gunfire; those who suffer from mental depression; those who have uncontrolled allergies; those with parents who have recently separated/divorced, or who have moved in with mom or dad's new significant other (or vice versa). Kids who have made bad choices (to drink, take drugs, have unprotected sex leading to teen pregnancy) have more chaotic lives than other kids; so do kids who are being abused. I am sure you can think of other examples too, but the number of kids in a school who are dealing with these issues is not considered when computing a school's SCI.
A parent who walks into their child’s kindergarten class on the first day of school can often spot the kid with the highest "chaos" score before even knowing any of the kids’ names. It is the boy who interrupts the teacher 6 times in the first 15 minutes, as she tries to get her 20 new students settled in a circle on the rug, or the girl who sits down on the rug, closes her eyes, and covers her ears, refusing to speak with anyone all day. These kids are going to need extra resources if they are to be successful in school, and having too many of them in any one school is going to pose a severe challenge to the school, one which does not show up on the SCI, and is not figured into the "similar schools" comparison. Schools which screen out kids with a high "chaos factor" can end up with a population of kids who look "similar" on an SCI table, but who are in fact far easier to educate, than the kids at another school.
Okay, I know I'm dreaming here, and there is no way to even get this kind of very personal information about kids and their families, let alone codify it into some kind of measurable chaos factor, and I realize that a student's "chaos score" might change from one year to the next, or even from one week to the next. But that doesn't mean that the chaos factor does not exist - I think it does. My point is, just because a table of "factors" indicates that two schools are facing "similar" challenges, or "similar numbers" of challenges, does not mean that the schools are really comparable in any meaningful way. Just as each child is unique, and comes to school with a unique set of advantages and disadvantages, so too is every school population unique. At best, the similar schools ranking is a crude measurement, one which compares schools which have some (but far from all) factors in common. It is one more statistic to look at when evaluating a school, but it would be irresponsible to use it as the sole, or even the most significant, measure of a school's success or failure.
Labels: Special Education