Here's a relevant quote from Managing the Non Profit Organization, written by Peter Drucker:
"Most of our current reporting systems don't reveal opportunities, they report problems."
Too often the data reviewed by organizations is more of an autopsy than a diagnosis. While the results of an autopsy shows how someone died it doesn't always or necessarily lead to any specific actions that can be taken to prevent additional deaths. This point is a refrain cited earlier in a Blog post that featured the poem, The Ambulance and the Valley, in which townspeople spend valuable resources on ambulances to pick up all of the people who drive off the cliff overlooking the valley instead of simply investing funds (far less than the money required for ambulances) in making a better fence as a deterrent to further casualties.
Another concern (which was the source of a previous Blog post) is the prospect of schools becoming so overwhelmed with collecting data that they become a DRIP school - Data Rich, Information Poor. In order to leverage success we must be capable of converting data into informed instructional decisions. That means looking for a difference that makes a difference. That begins with making sure that you have identified the right metrics. Peter Senge, author of Schools that Learn, offers the following point to consider:
"We tend to think that we believe what we measure, but it's more likely that we measure what we believe."
Finally, a reference from Passion for Excellence, written by Tom Peters and Nancy Austin:
"The number one sin (in terms of development or change) is the excessive quantification of the imponderable."