Community Action Agency Board Members Toolkit in a Nutshell

CHAPTER FIVE. What is ROMA?

A. What is ROMA?

1. What does “ROMA” stand for?
    ROMA stands for Results Oriented Management and Accountability.

 2. What is the purpose of ROMA?
    ROMA is a set of principles and tools to guide CAA program planning, operations and reporting. The intent of ROMA is to help CAAs to produce clearer and stronger results and to provide ways of measuring those results and reporting on them.

 3. Is ROMA a new way to address poverty in our communities?
    No. ROMA is not an anti-poverty strategy in and of itself, but is a new way of measuring the results of our anti-poverty work by measuring the changes that occur as a result of the services we provide. ROMA is a set of management principles and tools to measure results or “outcomes” for individuals, families and communities. Community Action Agencies (CAAs) must still figure out the causes and conditions of poverty and how to eliminate or reduce those causes and conditions in their local communities.

4. Why do we have to use ROMA?
    In 1993 Congress required all Federal agencies and programs to produce strategic plans and ways to measure results. In 1998, Congress amended the Community Services Block Grant legislation (CSBG) to mandate ROMA or some comparable system for CAAs. The Federal Office of Community Services (OCS) administers the CSBG program. There is no “comparable system,” therefore ROMA is the OCS required system for complying with the Federal law.

 5. Who developed ROMA?
    In 1995, the Office of Community Services (OCS) set up the Monitoring and Assessment Task Force (MATF). It is made up of 50 representatives of the Community Action Agency (CAA) network from around the country. MATF was charged with developing ways for CAAs to change the focus of their work from delivering services to ending poverty and to develop methods for measuring the results of this work.

6. Will ROMA last, or is it just another passing fad? 
    ROMA is the system for the foreseeable future. ROMA has been endorsed or adopted by all of the national organizations that work with the CAA network including the National Community Action Foundation (NCAF), the national Community Action Partnership, the National Association of State Community Services Programs (NASCSP) and by the HHS Office of Community Services.

B. THE SIX ROMA GOALS

7. What are the national goals for use by CAAs?

     There are six ROMA goals developed by the MATF and adopted by OCS for use by all CSBG “eligible entities,” most of which are CAAs. They are:

The Six ROMA Goals:

Using Logic Models for a Program, Strategy or Goal

   A logic model lays out the purpose, goals, activities, results desired, and measurements for an entire strategy or program. Typically these are laid out on one page in adjacent columns. It is a useful way to visually see the relationships between activities and outcomes. The logic model, developed to improve the ability to evaluate the results of program, went into widespread use in human services in the 1980’s. The OCS Demonstration Partnership Program required development of a logic model for each of the 100 projects they funded. Examples of logic models for family development, minority male, micro-business and other programs are included in the evaluation reports on those projects that are available from OCS.  The Early Head Start Program also requires development of a logic model.

Specific Tools: Scales, Ladders and Surveys

Scales and ladders are a way of measuring incremental change in individuals, families, communities or agencies. One familiar ROMA scale and ladder system currently in use measures a participant/client, agency or community as they progress from “In Crisis”, to “At Risk”, then “Stable” and then, hopefully, to “Safe” and then to “Thriving”. For example, a family may be “in crisis” because they are homeless. They get a temporary job and move into an over-crowded or unsafe apartment, so they are no longer homeless but still “at risk.”  Over time, with assistance from the CAA, their income increases and they obtain a more appropriate apartment - so they are now “stable.” As the family moves from up the scale or up the ladder from “in crisis” to “at risk” to “stable,” the CAA records the families progress and the CAA's efforts to help the family make this progress - and reports each step under a ROMA goal. Another useful measuring tool is a survey, a series of questions that provides feedback from a group of respondents. Like the customer satisfaction surveys used in restaurants or hotels, the survey ask a series of questions about what happened to you and how you feel about it. The results of the survey provide useful management information.

 F. MEASURING YOUR PROGRESS TOWARD ROMA IMPLEMENTATION

19. How can we assess our progress and ensure we are moving in the right direction toward full use of ROMA?

    The change to ROMA will take most agencies several years to complete. There are four basic ways to measure your progress. They include: 

G. ROMA. What is an outcome and how do I develop one?

Outcomes and results:

    Generally, to identify an outcome you have to look outside the agency to the program participants and the community. If you can measure it by watching the staff, it is probably an output.

     This sections discusses:

 1. Conceptual approaches to developing of the CONTENT of outcome measures.

     What are the types of outcomes?

     Because of the way G.P.R.A. and ROMA and Congressional mandated outcomes are coming at you, the selection of specific measures to be used for reporting at the local level or within a state is often presented as the starting point for discussion. At first glance, that appears to be the challenge. This probably makes the implementation process more difficult, because the discussion assumes the rest of existing program operations are a given and that only one thing – reporting measures – must be developed or changed. Wrong and wrong again.

    Whether you are looking at an agency or a program, changes in any one part of the system are going to precipitate changes in other parts of the system. If we change the measures, then everything above them (e.g. goals, objectives, strategies, activities) is going to be affected as well. The outcome measures selected become both crucial descriptor of but also drivers of program strategy, because in order to produce the desired result you have to use a specific strategy that affects that measure. Put another way, Julie Jakopic -- formerly of NASCSP -- says that “Because we end up doing what we measure, we need to choose carefully how we measure what we do.”  

    So we must consciously broaden the discussion about implementation of the new reporting system to also look at the conditions in the community, to include our theories about why the society works the way it works, and to include a review of the strategies we use to change it.  I know that this expansion of awareness takes time and many people resist doing it, because helped I peel this onion over about a three year period with USDA rural development programs. Any expansion of discussion to this broad range of topics immediately provokes resistance, especially from the defenders of the status quo. To them, an effort at large scale change implies that what we have been doing is somehow wrong, that we are guilty of not doing what we were supposed to be doing, and so on. One element of change management is to ELIMINATE RESIDUAL GUILT about changing something. This is done by the leaders who say things like:

“What we did was not wrong, it just was what we were doing. We did the best we knew how to do at the time, now we are looking for something else to try. Don’t feel bad about the past – feel good that we learned from it. Now, it is time to move on." 

    There are several ways to go about developing new measures of results and outcomes. You may use more than one approach, but it is useful to unravel them and look at the different assumptions on which they are based. And, no matter which way you start, you are probably going to wind up moving to a review of most of your program strategies.

A. Inductive approach from existing program operations. Start with what you’ve got, usually program outputs, and see how far you can stretch toward a description of outcomes. In working with the U.S.D.A. Rural Development mission areas, John Johnston and I assisted several working groups (housing, business, utilities) in development of results measures. We found that we could construct usable outcome measures (1) by starting with the programs existing activities, milestones and output measure(s), and (2) working outward very slowly and carefully toward the family and community, and (3) tracing every step, (4) to make sure there was a powerful link between every step. This was an inductive strategy -- to build what the evaluators call a logic model -- that starts with the program and moves outward by inches. And, it inevitably takes you into questions about basic program strategy.

B. Deductive approach from a plan. Start with community conditions and a vision mission and goals. This is the approach used by The Rensselearville Institute. Construct a far-reaching plan, identify your goals, and then figure out how to determine if the goal is achieved. Ask yourself: “How would I know if this goal had been achieved?  How would I measure it?”

C. Use a deductive approach that starts with social indicators and tries to bring it down to connect with a program. In recent years, approaches such as the Oregon Compact have used social indicators (the Oregon Benchmarks) as a way to focus attention on a subject area. A large-scale citizen planning effort led to the selection of the 259 measures, which were then boiled down into a set of about 20 high-priority measures. They are used to focus attention on a topic, e.g. “Let’s all do what we can to reduce teen pregnancy.”  But they are not used as a way of measuring performance of individual programs – they are used as signals that this is an important topic and people should be working on the issue. This also seems to be the approach being used by some of the Empowerment Zones and Enterprise Communities (EZ/EC initiative), but some EZ/EC’s appear to have confounded both social indicators and benchmarks. It is hard to figure out how changes in the small populations covered by the zones are going to show up in social indicators.

    Remember, in the 1970's both the United Way and the H.H.S. (then HEW) made a serious effort to use social indicators to measure progress in the social programs they funded – and they abandoned it. It did not work because most social indicators are the consequence of dozens or hundreds of factors that are working together to produce that result.  The unemployment rate, for example, is impacted by interest rates in Thailand, high-school graduation rates in Japan, migration patterns in Mexico, technology patents in Germany, and on and on. Conclusion. DO NOT adopt social indicators (unemployment rate, crime rate) as outcome measures for any of your efforts.

    The H.H.S./O.C.S. Monitoring and Assessment Task Force (M.A.T.F.) is suggesting you use social indicators only as descriptors of the CONTEXT of your work, i.e., the unemployment rate in the community may influence your ability to place people in jobs, but you should not assume responsibility for tying to change the unemployment rate per-se.

D. Customer satisfaction measurement. You can make a pretty good argument that our focus on program outputs is mis-focused because it gives too much credence to what we as program managers think the service “is”, and that in a service business the only thing that really matters is what the customer thinks has happened. Most service businesses  –airlines, hotels, restaurants, banks, doctors, accountants – ask their customers what they think and give them ways to give them systematic feedback through mail-in or drop-in-the-box survey forms. Most Federal agencies now do the same thing, including the Social Security Administration and the DOL employment and training programs. Most cities and school districts also measure customer satisfaction.  (The author is highly opinionated on this subject and in the interest of full disclosure should report that he develops and conducts customer satisfaction surveys for Head Start programs.)

E. Expert Observations.  Use professional opinion of the school counselor, teacher, psychologist, or social worker. You are in effect relying on the credibility and authority of this expert to validate the measurement. One Head Start Director in Virginia asks all the school counselors “Do you see any difference between the Head Start children and the others.”  And they say “Yes.” And he says “Would you write me a letter on your letterhead stating that.”  And they do, and so he has a stack of letters validating the program that he carries in his briefcase to show to people. Clever!

F. Combine Customer Observation with Expert Observation. Develop scales and ladders to connect what you do with the results produced. One of the useful tools adapted for use in ROMA are scales and ladders.

    These rely on the combined judgment of the participant and the staff person (the expert). They are a good tool to use to measure incremental change over time. There are several examples on the ROMA web site, at www.roma1.org

     You can use the scale and ladder approach to measure just about anything. Put the worst case scenario at the bottom, the best case at the top. Create intermediate steps that show progress from the worst to the best. Eureka – a scale and ladder!

 End-of-Chapter Quiz

Answers to Chapter Five Quiz

Toolkit Home      Chapter 6