 |
Phone: 510-339-3801 Fax: 510-339-3803 E-mail: jmasters@cencomfut.com |
Mailing Address: Center for Community Futures P.O. Box 5309 Berkeley, CA 94705 |
                 |
|
|
|
CHAPTER FIVE.
About ROMA
A. What is ROMA?1. What does “ROMA”
stand for? 2.
What is the purpose of ROMA? 3.
Is ROMA a new way to address poverty in our communities? 4. Why do we have to use ROMA? 5. Who developed ROMA?
B. THE SIX ROMA GOALS
7. What are the national goals for
use by CAA’s?
These
are agency-wide goals. Under
these, each CAA lays out its specific goals for programs or strategies. 8.
CAA’s
are all very different, so how can we all use the same goals? 9.
Are the six ROMA goals just for CSBG funded programs and services or for
the entire agency and all of it’s services? 10. How can we organize all of our
programs and services under the ROMA goals? C. OUTCOMES AND INDICATORS
11. How
do we plan for Results? 12. What is an Indicator? 13.
How do we go about designing the Outcomes we want? 14.
Where can I go for help in developing or measuring outcomes or
results? D. REPORTING15.
Is ROMA just a new way of reporting the services we provide? 16.
Does this mean we can no longer deliver or report on emergency
services? 17.
We find some of the data we have collected in the past to be very
useful. Do we have to abandon the data collection we already do? E. APPROACHES to IMPLEMENTATION 18.
What are the most common approaches to
ROMA?
A.
Agency-wide management framework: two approaches
a) The Virtual Outcomes College is using the Drucker Foundation’s Self Assessment Tool in their ROMA Peer-to-Peer training course. The Virtual Outcomes College (with funding from OCS) has adapted the Drucker Self Assessment Tool for use by CAA’s. It is a comprehensive framework of vision setting, mission review, goal setting and reporting. b)
The Rensselearville Institute in New York has developed a planning
and management system for use by CAA’s that has been adopted by CAA’s in
about a dozen states. The state or national CAA Association can provide
contact information on both of these systems. Using
Logic Models for a Program, Strategy or Goal. Specific Tools: Scales, Ladders and Surveys
Scales
and ladders are a way of measuring incremental change in individuals,
families, communities or agencies. One familiar ROMA scale and ladder system
currently in use measures a participant/client, agency or community as they
progress from “In Crisis”, to “At Risk”, then “Stable” and then,
hopefully, to “Safe” and then to “Thriving”.
For example, a family may be “in crisis” because they are
homeless. They get a temporary job and move into an over-crowded or
unsafe apartment, so they are no longer homeless but still “at risk.”
Over time, with assistance from the CAA, their income increases and
they obtain a more appropriate apartment - so they are now “stable.” As
the family moves from up the scale or up the ladder from “in crisis” to
“at risk” to “stable,” the CAA records the families progress and the
CAA’s efforts to help the family make this progress - and reports each
step under a ROMA goal. Another useful measuring tool is a survey, a series
of questions that provides feedback from a group of respondents. Like the
customer satisfaction surveys used in restaurants or hotels, the survey ask
a series of questions about what
happened
to you and how you feel about it. The results of the survey provide useful
management information. F. MEASURING YOUR PROGRESS TOWARD ROMA IMPLEMENTATION19. How can we assess our progress and ensure we are moving in the right
direction toward full use of ROMA?
| |||||||||||||||||||||||||||||||||||||||||
| Inputs | Activities | Outputs | Outcomes | |
| Traditional/past | 40 | 10 | 40 | 10 |
| What GPRA wants | 10 | 10 | 10 | 70 |
| What Purists want (tell us what we bought) | 10 | 0 | 0 | 90 |
| Where we are in transition | 30 | 10 | 40 | 20 |
Put another way, the purists would have the Federal
agencies go to a Zero Based Budgeting System (used by President Carter).
“Assume your budget is zero. Tell
us what results you would produce and how much money these results will
cost.” This has recently been re-emphasized in the Bush
Administration as part of the President’s Management Agenda (see the OMB
web site at http://www.whitehouse.gov/omb/budintegration/pma_index.html
This lays out his ideas for changing Federal agencies and enhancing outcomes measurement.
We are in a transition phase, and many agencies are having a difficult time
in improving their results and outcomes.
One reason for this is that instead of changing the program model and
changing the assumptions, strategies and work program descriptions, most
funders have instead added a new template on top of existing requirements.
Most funders have left program formats, strategies, and budgets in
place, and added a new reporting form.
The approach of leaving all existing elements in place and adding a
new section on outcomes may make people believe that this is “just another
reporting” requirement. This
may reduce the urgency of developing the results and outcome measures, or of
changing strategies to produce better outcomes.
Given the enormous complexity of coming up with results
measures for most human development programs, this is understandable.
The task is further complicated by the current status of social
science, which does not offer immediate solutions to our dilemma (See
below).
So if we have boiled ROMA down to its essential element –
an increased focus on outcomes. In
the process of reducing ROMA to its core elements we have also removed any
bias toward a specific type of outcome or a specific intervention strategy
is removed or a specific method of measurement.
What, then, is left in the pot?
What is left is the urgent need to develop new and better ways of describing the changes produced in individuals, families, communities as a result of our work, and to develop better ways (strategies) of producing desired changes. You might think that having de-constructed this Christmas tree and boiled off the decorations that what is left would be simple to deal with. Unfortunately this is not the case. Describing outcomes is a huge challenge. There are several reasons for this.
Humans tend to
confuse ideology (motives or good intentions) with results.
This is a classic in human services – people want to be judged
for their good intentions not what changes they produce.
Most Federal statutes are a product of compromise. They
have numerous internal contradictions built into them.
(Cite GAO reports 1986 and 1987).
The muddled legislation represents compromises and messy
combinations of theories, causes, strategies, programs, results,
activities, participant groups, etc. You can NOT get from most Federal
statutes to a coherent group of results unless you do a lot of “social
construction” in the middle. Logic
models can help us address these problems by creating a framework of
goals, activities results and measures.
Until the 1980's, the assumption was that you described
your program in the grant application and then ran it that way for the
whole year. This enables
funders and evaluators to see which of your activities worked and which
did not. Starting with the
quality movement that empowered from-line staff to make changes every
day to meet customer needs, the static program model began to erode.
CQI, TQM and other approaches now legitimize constant changes and
learning as time passes. This
makes it difficult to measure efficiency or cost-effectiveness.
Results measurement requires great clarity about “what is
to be done” but – in many agencies goals are often vague and open
ended. Even if the goal is
clear, we usually do not know what is to be done in any specific
situation until there is a (private) interaction between the staff
person and the program participant and they reach an agreement about
what the participant will do and what the staff will do.
So the “what” varies significantly from participant to
participant, and is unknown until the staff and that person reach an
agreement.
Much of what we seek is to promote a
social movement for social justice reasons (greater equity, fairness)
which are difficult concepts to measure, and take decades to accomplish.
Now we have identified some of the small problems,
let’s look at the BIG problem.
Limits of Social Science. Most sociology, cultural anthropology and political science theory is big theory that applies to large groups of people. It is not a small theory that is true for every individual. You can not apply big theory to individuals.
Education has been developing measurement tools for 100 years;
sociology for 75 years, and psychology has been developing tests for 50
years. These measurement tools
are developed incrementally and get better as time – usually decades –
passes. Many of these
measurement systems or tests have been “normed” on millions of people.
Most of them have to be administered by a trained person.
And, they still have rigid limits.
Most are univariate – they measure one trait.
If you try to use the test for anything outside that single trait, it
does not work. We read stories
weekly about the controversies of using tests in public schools.
If you have a program where you can use hard science like
physics and chemistry and engineering and micro-economics to measure
results, like WX., you can do a good evaluation. The Oak Ridge evaluation of
WX is an excellent example. Unfortunately,
most human development programs are not based on these sciences.
So how DO we create results measures? This is a process of social construction. We must create a miniature theory that includes:
* what the condition is
* What
the cause is
* What strategies will change the
cause and thereby change the condition
*
This is an iterative process; we try something and then
improve it. It takes years to
develop good outcome measures. Also,
just to be clear, I think that scales and ladders ARE a good tool for
CAA’s, because they enable you to measure incremental change over time.
This is a challenging process.
If every local program has a slightly different condition or
strategy, how do we compare them? If
we standardize measures on a nationwide basis, then how to we respond to
unique conditions? There is a major debate taking place at the national level
about how we should add up the work that all CAA’s nationwide are doing.
Under the current system, CAA’s select their own goals and
outcomes. The Bush
Administration has proposed that a dozen or so measures be used for all
CAA’s, and that each CAA will assume a proportional responsibility for
producing on that measure.
So, the development of results and outcome measures locally
requires a LOT of conversation, analysis and thinking about why people are
poor, what we can do about it, and how we measure it.
The Community action network is now nine years into this process
(remember we started in 1994) and making good progress.
Every other Federally funded program is going through the same
process. We are ahead of most
of the rest. Keep up the good
work!
The next section will provide ideas on ways to develop
results and outcome measures at the local level.
Outcomes and results:
* help to explain and illustrate the multi-year goals.
* a
*
Generally, outcomes can:
* be measured.
* u
* describe the desired condition/s that will exist and how that is
different from the existing condition/s.
|
“I will be able
to show that this goal has been achieved when . . .” |
Generally, to identify an outcome you have to look outside
the agency to the program participants and the community.
If you can measure it by watching the staff, it is probably an
output.
What are the types of outcomes?
Because of the way G.P.R.A. and ROMA and Congressional mandated outcomes are coming at you, the selection of specific measures to be used for reporting at the local level or within a state is often presented as the starting point for discussion. At first glance, that appears to be the challenge. This probably makes the implementation process more difficult, because the discussion assumes the rest of existing program operations are a given and that only one thing – reporting measures – must be developed or changed. Wrong and wrong again.
Whether you are looking at an agency or a program, changes in any one part of the system are going to precipitate changes in other parts of the system. If we change the measures, then everything above them (e.g. goals, objectives, strategies, activities) is going to be affected as well. The outcome measures selected become both crucial descriptor of but also drivers of program strategy, because in order to produce the desired result you have to use a specific strategy that affects that measure. Put another way, Julie Jakopic -- formerly of NASCSP -- says that “Because we end up doing what we measure, we need to choose carefully how we measure what we do.”
So we must consciously broaden the discussion about implementation of the new reporting system to also look at the conditions in the community, to include our theories about why the society works the way it works, and to include a review of the strategies we use to change it. I know that this expansion of awareness takes time and many people resist doing it, because helped I peel this onion over about a three year period with USDA rural development programs. Any expansion of discussion to this broad range of topics immediately provokes resistance, especially from the defenders of the status quo. To them, an effort at large scale change implies that what we have been doing is somehow wrong, that we are guilty of not doing what we were supposed to be doing, and so on. One element of change management is to ELIMINATE RESIDUAL GUILT about changing something. This is done by the leaders who say things like:
“What we did was
not wrong, it just was what we were doing.
We did the best we knew how to do at the time, now we are looking for
something else to try. Don’t
feel bad about the past – feel good that we learned from it.
Now, it is time to move on.
There are several ways to go about developing new measures of results
and outcomes. You may use more
than one approach, but it is useful to unravel them and look at the
different assumptions on which they are based.
And, no matter which way you start, you are probably going to wind up
moving to a review of most of your program strategies.
A. Inductive approach from existing program operations.
Start with what you’ve got, usually program outputs, and see how
far you can stretch toward a description of outcomes.
In working with the U.S.D.A. Rural Development mission areas, John
Johnston and I assisted several working groups (housing, business,
utilities) in development of results measures.
We found that we could construct usable outcome measures (1) by
starting with the programs existing activities, milestones and output
measure(s), and (2) working outward very slowly and carefully toward the
family and community, and (3) tracing every step, (4) to make sure there was
a powerful link between every step. This
was an inductive strategy -- to build what the evaluators call a logic model
-- that starts with the program and moves outward by inches.
And, it inevitably takes you into questions about basic program
strategy.
B. Deductive approach from a plan. Start with community conditions and a vision mission and goals.
This is the approach used by The Rensselearville Institute.
Construct a far-reaching plan, identify your
goals, and then figure out how to determine if the goal is achieved.
Ask yourself: “How would I know if this goal had been achieved?
How would I measure it?”
C. Use a deductive approach that starts with social indicators
and tries to bring it down to connect with a program. In recent years, approaches such as the Oregon Compact have
used social indicators (the Oregon Benchmarks) as a way to focus attention
on a subject area. A
large-scale citizen planning effort led to the selection of the 259
measures, which were then boiled down into a set of about 20 high-priority
measures. They are used to
focus attention on a topic, e.g. “Let’s all do what we can to reduce
teen pregnancy.” But they are
not used as a way of measuring performance of individual programs –
they are used as signals that this is an important topic and people should
be working on the issue. This
also seems to be the approach being used by some of the Empowerment Zones
and Enterprise Communities (EZ/EC initiative), but some EZ/EC’s appear to
have confounded both social indicators and benchmarks.
It is hard to figure out how changes in the small populations covered
by the zones are going to show up in social indicators.
Remember, in the 1970's both the United Way and the H.H.S. (then HEW)
made a serious effort to use social indicators to measure progress in the
social programs they funded – and they abandoned it.
It did not work because most social indicators are the consequence of
dozens or hundreds of factors that are working together to produce that
result. The unemployment rate, for example, is impacted by
interest rates in Thailand, high-school graduation rates in Japan, migration
patterns in Mexico, technology patents in Germany, and on and on. Conclusion. DO
NOT adopt social indicators (unemployment rate, crime rate) as outcome
measures for any of your efforts.
The H.H.S./O.C.S. Monitoring and Assessment Task Force (M.A.T.F.) is
suggesting you use social indicators only as descriptors of the CONTEXT of
your work, i.e., the unemployment rate in the community may influence your
ability to place people in jobs, but you should not assume responsibility
for tying to change the unemployment rate per-se.
D. Customer satisfaction measurement. You can make a pretty good argument that our focus on program
outputs is mis-focused because it gives too much credence to what we as
program managers think the service “is”, and that in a service business
the only thing that really matters is what the customer thinks has happened.
Most service businesses –airlines,
hotels, restaurants, banks, doctors, accountants – ask their customers
what they think and give them ways to give them systematic feedback through
mail-in or drop-in-the-box survey forms.
Most Federal agencies now do the same thing, including the Social
Security Administration and the DOL employment and training programs.
Most cities and school districts also measure customer satisfaction. (The author is highly opinionated on this subject and
in the interest of full disclosure should report that he develops and
conducts customer satisfaction surveys for Head Start programs.)
E. Expert Observations.
Use professional opinion of the school counselor, teacher,
psychologist, or social worker. You are in effect relying on the credibility
and authority of this expert to validate the measurement.
One Head Start Director in Virginia asks all the school counselors
“Do you see any difference between the Head Start children and the
others.” And they say
“Yes.” And he says “Would you write me a letter on your letterhead
stating that.” And they do, and so he has a stack of letters validating the
program that he carries in his briefcase to show to people.
Clever!
F. Combine Customer Observation with Expert Observation.
Develop scales and ladders to connect what you do with the results
produced. One of the useful
tools adapted for use in ROMA are scales and ladders.
These rely on the combined judgment of the participant and the
staff person (the expert). They
are a good tool to use to measure incremental change over time.
There are several examples on the ROMA web site, at www.roma1.org
You can use the scale and ladder approach to measure just about
anything. Put the worst case
scenario at the bottom, the best case at the top.
Create intermediate steps that show progress from the worst to the
best. Eureka – a scale and
ladder!
A. For Head Start. Select measures of outcomes and results from IM’s, and the many research studies. For IM’s and other Head Start publications, go to the Head Start publications center at www.headstartinfo.org/searc.htm
IM 03. Initial Guidance on New Legislative Provisions on Performance Standards, Performance Measures, Program Self-Assessment and Program Monitoring. Issued in 2000. Copy Appendix A and B at http://www.headstartinfo.org/publications/im00/im00_03.htm
IM 18. Child Outcomes, Performance Measures, Program Self-Assessment: Using Child Outcomes in Program Self-Assessment Be sure to also copy Appendix A which has the Child Outcomes Framework. http://www.headstartinfo.org/publications/im00/im00_18.htm
They frequently change the specific links given below, but you can always keep track by starting at the ACF homepage at http://www.acf.dhhs.gov/ then go to ‘Research and Publications’, then go to ‘Head Start Bureau Research’ then to ‘Ongoing Research’ For example, FACES is at www.acf.dhhs.gov/programs/core/ongoing_research/faces/faces_intro.html
The Head Start Quality Research Consortium has several reports on program practices and outcomes. Check it out at: www.acf.dhhs.gov/programs/core/ongoing_research/qrc/qrc.htm
The site for the Impact Evaluation of Head
Start is at
www.acf.hhs.gov/programs/core/ongoing_research/hs/impact_intro.html
and the impact evaluation for Early Head start is at:
www.acf.dhhs.gov/programs/core/ongoing_research/ehs/ehs_intro.html
The Head Start/Public Schools Early Childhood Transition
Demonstration program reports are at www.acf.dhhs.gov/programs/core/pubs_reports/hs/transition_intro,html
For Head Start and Preschool, use the County of San Bernardino
County, Human Service System, Preschool Services Department Measures. (909.387.2375) The
Florida R&D projects have new measures.
The California Head Start Association and the State Dept of Education
negotiated a mutual set of school readiness measures and other outcomes,
called the D.R.D.P.+ Both
are seeking the same desired results.
And, use logic models.
Most of the Community Development Corporations, including those funded from
the O.C.S. discretionary funds, are involved in the Success Measures Project funded by the McAuley Institute and
operated by the Development Leadership Network, Baltimore, MD.
They created working groups in ten locations around the country to
try to create outcome measures for the complex community-development and
community- building projects. They
struggled with all the same issues address at USDA, HUD and H.H.S. efforts,
but did not use much of the experience from the other agency efforts.
It was more of a “whole cloth” re-invent the wheel type of
effort, which of course can sometimes produce fresh new tools.
The topics areas in which they worked are: Housing, Business and Job
Development, community participation, levels of hardship, and the
relationship between funders and implementors.
Their initial publication is “Success Measures Guide Book version
1.0”. The Development Leadership
Network, 685 Centre Street, Boston, MA 02130, 617-971-9443.
Their FAX 617.971.0778. Contact
info@developmentleadership.net,
and their guidebook is on their web site at:
www.developmentleadership.net
In the meantime, a very useful tool is already available from the
Virginia CAA’s. They
developed and piloted this in five agencies.
It is the “Community Matrix Focusing on Neighborhood Organization
and Civic Capital.” It is a
type of scale-and-ladder that shows how you can help a community move from
dis-organization to higher levels of functioning.
This is VERY useful – it is a major contribution to our field.
Thanks to the Virginia Head Start Sponsors for developing this, and
thanks to Ted Edlich and others at Total Action Against Poverty for making
it available to others. WRITE
to: Ted Edlich; TAAP, 145
Campbell Avenue, Roanoke, VA 24011. The
Illinois CAA Association and Missouri Associations are also working on these
measures. Pennsylvania has some
as well. Ask your staff to
collect some of these and look them over.
The North Central Regional Central for Rural Development, Iowa
State University, also has a useful publication Working
Toward Community Goals: Helping Communities Succeed that combines both
community development strategy and reporting of results.
It came out of collaboration between the Aspen Institute, Ford
Foundation and USDA. Their web
page is: http://www.ag.iastate.edu/centers/rdev/RuralDev.html
Also, check out the material produced by the Northern California
Council for the Community at http://www.ncccsf.org/
Their publications are Goals and Success Indicators of Healthy and Self-Sufficient Communities, and
is Bay Area Partnership:
Building Health and Self-Sufficient Community for Economic Prosperity,
Status Report. This is
based on use of social indicators and is similar to Oregon Benchmarks, but
brought down to a community level.
A VERY useful publication to use to look at all the Federal block
programs in terms of their purpose, structure, relationship to states is the
GAO publication, Grant Programs: Design Features, Shape, Flexibility, Accountability and
Performance Information.
Our whole universe is in motion! All Federal agencies are changing their program and reporting structures. You are being asked to “coordinate” and “plan with partners” but Federally funded program structures are changing and will be with us for several more years.
C. The United Way
as an excellent source of info on how to measure human service programs.
(Tab 18)
http://national.unitedway.org/outcomes/publctns.htm
D. The bibliography
in this publication lists the International City Management Association and
other good sources of methods for developing results and outcome measures.
E. Mix and Match. Most agencies use a mixture of the approaches described
above. It is important to be
clear about which method you are using, otherwise you may find yourself
where you think the discussion is about the content of a measure when it is
actually about the logic system you are using to develop the measure.
There is an argument to be made that the process you create for this change effort is much more important than the content you create, because the process will be used to modify, evolve, revise and change the content for the next several years. There is no such thing as developing results measures and using them forever. Next, we provide four examples to illustrate some of the possibilities in developing outcome measures.
1.
Every Program Does Its Own Thing.
The funders take a hands-off approach and let each program develop
whatever it wants.
Advantages.
* M
* P
Disadvantages.
* Unable to relate each local program’s activity to the activity of others.
*
* Slow – everybody has to reinvent their wheel.
*
*
2. A Federal or State Agency Mandates the System.
Advantages.
* F
*
* M
Disadvantages.
* C
*
*
* Resistance to “Federal or state bureaucracy.”
3.
It Is A Group Effort, But Factions Form among the local programs,
and one group is allowed to win.
One faction consists of the “lumpers” who want to combine a lot
of types of activity into very general measures, the other consists of the
“splitters” who want to develop detailed and very specific measures.
(These types of factions also form among linguists, and among
ethnologists who study race.) The
two factions reach a stalemate. One
faction outlasts the other and “wins” and imposes their definitions on
all programs. The losers have a
serious case of sour grapes and evade use of the new system as much as
possible.