Data Disorientation

"We use data like a drunk uses a lamp post - for support, not illumination"  Robert Waterman, The Renewal Factor

Public schools, particularly in New York, where the state has been awarded $700,000,000 as a recipient of the federally funded competitive grant entitled Race To The Top, have recently embarked on yet another crusade, this one fueled and governed by data. I will be attending a conference this Friday devoted to topics emerging from the expectations and requirements of schools with respect to Race To The Top. My skepticism is not simply generated by the fact that our school will only receive $3,800 per year for each of the next four years to reach the standards enumerated in this program. Rather, my anxiety evolves from the fear that this is another well intended but misguided effort destined to fall short of its goal.

The campaign is predicated on a concern arising from a reservoir of statistics used as performance gauges. Who can argue with the objectivity and impartiality of numbers? While the sanctity of the arithmetic may not be questioned, the application and interpretation of the calculations should be the subject of critical examination. This contention hearkens back to another quote that has appeared in previous Blogs – “Not everything that counts can be counted, and not everything that can be counted, counts” (George Pickering).

Years ago, a professor of statistics and probability once informed me that, “Figures don’t lie, but liars can figure.” I’m not suggesting that educational reformers are unethical or dishonorable. Neither are they lacking in concern or commitment. They want to improve education, and that’s certainly a noble goal shared by all, but good intentions and sheer will-power are not enough to ensure success in any venture.

Technology now allows us to store, access, and analyze an unprecedented and incredible amount of data with extreme speed and efficient accuracy. However, the fact that we can analyze learners by shoe size, address, hand dominance, eye color, intelligence quotient, height, or any imaginable amount of measurables in addition to academic achievement does not necessarily promote our progress toward increasing performance levels. If you don’t know where you’re going, a faster car only gets you lost faster!

This motivation to move forward resembles the same compelling fervor that prompted the explosive increase in Math and Science courses in American public schools following the Russian’s launch of the first earth-orbiting satellite in 1957, or the surge in reports of research in Reading and the development of Reading programs after the 1983 publication of A Nation at Risk.  

Much has been said about the collection and interpretation of data in schools. Yet, it’s quite interesting that some of the most thought provoking statements that could be applied to the use of data in education relate to being drunk. The message included in this Blog addresses the problems of being intoxicated by data. This condition can be remedied by re-examining how we identify and interpret data.

Schools need to be driven by a mission communicated in language that inspires people to pursue a commonly held vision of success associated with teaching and learning. Data should therefore be utilized to illuminate the route to the desired vision. The inability to clearly agree upon a mission results in acting on data that may not be directly related to the school's real purpose. Schools often act like the drunk portrayed in the following story.

One night on my way home I spotted a drunken man on his hands and knees crawling around in the area beneath a street light.

"I lost my keys, can you help me?" he asked.

"Sure," I replied, "did you lose them around here?"

"No" he said.

“Where did you last remember having your keys?” I inquired.

“Over there,” he stated as he pointed to a building over 100 feet away, "I lost them over there by that old building."

I was puzzled by his response, so I asked him, "Then why are you searching under this street light?" "Well," the drunk said, "there isn't any light over there."

The keys we are searching for are the keys to effectiveness in schools. And we won't find them looking in the dark. Light must be shed upon the meaning and purpose of schools so an appropriate data base can be developed to create benchmarks along our journey, much like the mile markers beside long stretches of highway that inform drivers of their progress.

The first step in constructing an appropriate database is crafting a mission related to an accepted vision. This beacon will allow us to differentiate between necessary data and confusing, inconsequential data that may be interesting but not critical in leveraging success. Former Secretary of Defense Donald Rumsfeld once remarked about the excessive data confronting his department, "There's such a flood of information, it's like drinking out of a fire hose."

In their book, Reinventing Government, authors Osborne and Gaebler state: "Most reporting systems don't reveal opportunities, they report problems." Problems make for headlines and soundbites that attract sales of newspapers or ratings on television, but they can cloud our vision every bit as much as clarify our focus. Robert Waterman encourages readers of his book, The Renewal Factor, to “look for a difference in data that makes a difference.” To improve schools we must seek opportunities rather than focus on problems. This is analogous to the expression, "We need to play to win instead of playing not to lose." There is a big difference. One plan is action oriented and assertive while the other is reactive and defensive.

Step two, therefore, is to collect data by exploring opportunities as vehicles to transport the school toward the vision. For purposes of exploration, use a kaleidoscope to examine new and changing patterns, and a telescope to search distant opportunities, instead of a microscope that focuses upon minutiae. Look outside the traditional parameters we impose upon ourselves when engaged in school improvement efforts. Ask yourself – What really counts?

Unshackling the blinders that conspire to limit our opportunities can be productive. For example, a recent study conducted by John Jewkes and reported by Roy Rowan in his book, The Intuitive Manager, reveals that of 58 major twentieth century inventions (defined by their impact upon the daily lives of people), 46 of these discoveries were produced by an individual, a small firm, or somebody in the "wrong business." King Gillette was a cork salesman when he came up with the safety razor. George Eastman was a bookkeeper who changed the nature of photography. Two musicians invented Kodachrome. A veterinarian, John Dunlop, was the co-inventor of the pneumatic tire. An undertaker devised the automatic dialing system. A watchmaker seeking a solution to a brass fitting problem developed the concept of continuous casting steel.

Step three involves interpreting the data. Look for leverage points where inert data can be converted into user-friendly information that can be applied by practitioners as solutions. This technique must observe dissonant information as well as supportive data. Avoid the fate of the frog that was boiled. If you place a frog into hot water it will leap out, the same response that extremely cold water would provoke. However, if you place the frog in a pot of water that approximates the temperature of it's pond water you can then gradually heat up the water to the boiling point without the frog noticing the imperceptible change in temperature. Another view on data interpretation uses the example of a double loop feedback system. A thermostat is a typical single loop system. You set the gauge for the desired temperature, much like you establish goals for the school. Although the thermostat can direct the furnace to respond to a change in temperature it can not answer the question "Do we still want this same temperature three hours from now?" Constant interpretation of data will respond to the question, "Are we still on the right track?"

School improvement should not drive you to drink. By following the three suggestions indicated above you will be less likely to become data drunk: 1.) identify appropriate data, 2.) seek opportunistic data, and 3.) constantly interpret the data.