Interview: Change Evaluation – Is Your Initiative Working?
May 11, 2010
This morning on The Change Agent’s Dilemma radio show, I interviewed Maria Gajewski, Chief Assistant to World Savers at Changing River Consulting.
As an evaluator for nonprofits, Maria is an expert at determining whether an initiative is having its intended effect. On the show, she shared how evaluation works and when you would want to evaluate your change initiative. She also shares the tools she uses in the evaluation process.
Maria has provided a summary of today’s discussion for Enclaria listeners. Download the PDF here.
Listen here (30 minutes):
Be sure to visit the radio show page to listen to past episodes and subscribe to the show.
Make Measures Work
May 6, 2010
It’s been said, “What gets measured gets done.” Taken at face value, the statement means that if you want something done, then simply measure it. Anyone who has tried to implement a balanced scorecard knows that just because you measure something doesn’t mean it’s automatically going to change. There are a number of caveats and prerequisites to the well-worn platitude. The following are a few ways to make measures work. Read more
Four criteria for strategic measures
March 13, 2010
Yesterday I went to a meeting of the Association for Strategic Planning Atlanta, where some members of a team at Deloitte presented their process for strategic measure selection. One step in the process, after brainstorming measures for each strategic objective, was to put the measures through an evaluation funnel. There were four key criteria for selecting measures:
- Strategic. Does the measure track progress toward achieving the objective?
- Actionable. Will you be able to do anything about it? Will it help you make decisions?
- Operational. Is it something you are able to collect? Can people understand it and do they believe it?
- Economical. Is the cost of collecting the information feasible?
After measures made it through the evaluation funnel, they would be further whittled down until there were 1 or 2 measures for each strategic objective.
Would the measures for your change initiative make it through the funnel?
Use logic modeling to check your assumptions
February 13, 2010
Yesterday I attended a meeting of the Association for Strategic Planning Atlanta chapter. The presenter was Tom Chapel, Chief Evaluation Officer for the Centers for Disease Control and Prevention. His job is to make sure all the programs designed and implemented by the CDC actually cause the results they are intended to cause.
The example he used during the presentation was a program to reduce lead poisoning in children. In his core diagram were the activities done by the CDC and others in the community, such as Conducting Screenings, Identifying Children with Elevated Lead Levels, and Training Parents. These were all the things under the control of the CDC and its partners.
On the right side of the diagram were all the intended outcomes the activities were expected to influence, such as Changes to Cleaning Habits, and Reduced Lead Levels. One of the shorter-term results was a change in at-home behaviors of parents, and the longer-term effects were the health benefits. Each of the activities and outcomes were connected by arrows in cause-and-effect linkages. He called the diagram a logic model, since it showed the assumptions, or logic, behind how the program was expected to work.
The generic logic model looks like this:
The same type of diagram can be used to model the expected outcomes of organizational change activities (in fact the speaker mentioned that the CDC has done logic models for internal initiatives like diversity and leadership development). The Control side would contain all the activities you are implementing, such as various communication vehicles, incentives, and training programs. On the Influence side would be all the behaviors, attitudes and culture changes your activities are expected to elicit.
Once you create the logic model, you can test your assumptions. Will Activity A really cause Behavior B? What reinforcements might be required? What other factors might stand in the way? In his presentation, Mr. Chapel said to look for where the miracles are supposed to happen and unravel them.
The purpose of the logic model is to gain clarity about the relationship between activities and their intended effects. Over time, by measuring the activities and outcomes, you can test the logic model to see if it’s true.
What are your assumptions about how your change initiative will create its intended outcomes?
Case Study: A Fizzled Balanced Scorecard Implementation
March 5, 2009
In the history of Balanced Scorecard implementations and organizational change in general, stories of projects falling by the wayside are numerous. Even organizations that have had great success and demonstrated breakthrough results have fallen away from the methodology, either due to a change in leadership or from the initiative growing stale. In the case of The Jel Sert Company, a mid-size food manufacturer based in West Chicago, Illinois, the project never quite gained the traction it needed to sustain itself, and the leadership team made the conscious decision to end it. Read more
Shampoo lather and other leading indicators
January 19, 2009
My friend Candace said to me one day, “Did you know that shampoo doesn’t need to lather in order to do its job?” She told me that when she was growing up in South Africa, shampoo there did not lather. When she moved to the U.S., she was surprised at the foaming action exhibited by all shampoos.
Evidently, we need some kind of proof that the shampoo is working. We need reassurance that a product is doing its job when outcome indicators are not immediately available – in other words, we need leading indicators to tell us that we are on track to attain the outcome we expect. The more lather, the cleaner the hair, right?
I’ve always had trouble with the concept of leading indicators, because the logical, literal side of my brain tells me that all measures are lagging because some event has to occur before something can be measured. However, the leading or lagging description does not tell us the timing of the measure to the event; it tells us the timing relative to each other. Leading indicators are predictions of future lagging measures. If something is a leading indicator, it shows whether you are progressing toward the end result – the lagging measure. So, for example, if the lagging (outcome) measure is the amount of weight you’ve lost, then the leading (process) indicator may be the number of times you go to the gym, or how many calories you consume per day.
When implementing organizational change, it is important to have and communicate short term wins. One way to do this is with leading indicators. By making the shampoo lather, you can see it is working without having to wait for the final evidence.
Inquiry: What is your shampoo lather?
Don’t miss a thing!
Subscribe
in a ReaderSubscribe
via E-mail
-
