Phone: 510-339-3801
Fax: 510-339-3803
E-mail: jmasters@cencomfut.com
Mailing Address:
Center for Community Futures
P.O. Box 5309
Berkeley, CA  94705
Home Community Development Toolkit for Boards Head Start Social Values Poverty Work Search NCPES Agency Management Opinions by Jim Conference Planning Microbusiness Events Web Site Design Our People Order Form Rooting Out Poverty

 

CHAPTER FIVE. About ROMA

A. What is ROMA?

1.  What does “ROMA” stand for?
    ROMA stands for Results Oriented Management and Accountability.

2.  What is the purpose of ROMA?
    ROMA is a set of principles and tools to guide CAA program planning, operations and reporting.  The intent of ROMA is to help CAAs to produce clearer and stronger results and to provide ways of measuring those results and reporting on them.

3. Is ROMA a new way to address poverty in our communities?
    No.  ROMA is not an anti-poverty strategy in and of itself, but is a new way of measuring the results of our anti-poverty work by measuring the changes that occur as a result of the services we provide. ROMA is a set of management principles and tools to measure results or “outcomes” for individuals, families and communities. Community Action Agencies (CAA’s) must still figure out the causes and conditions of poverty and how to eliminate or reduce those causes and conditions in their local communities.

4. Why do we have to use ROMA?
    In 1993 Congress required all Federal agencies and programs to produce strategic plans and ways to measure results.  In 1998, Congress amended the Community Services Block Grant legislation (CSBG) to mandate ROMA or some comparable system for CAA’s.  The Federal Office of Community Services (OCS) administers the CSBG program.  There is no “comparable system,” therefore ROMA is the OCS required system for complying with the Federal law.

5. Who developed ROMA?
    In 1995, the Office of Community Services (OCS) set up the Monitoring and Assessment Task Force (MATF).  It is made up of 50 representatives of the Community Action Agency (CAA) network from around the country.  MATF was charged with developing ways for CAA’s to change the focus of their work from delivering services to ending poverty and to develop methods for measuring the results of this work.


6.
Will ROMA last, or is it just another passing fad?
   
ROMA is the system for the foreseeable future.  ROMA has been endorsed or adopted by all of the national organizations that work with the CAA network including the National Community Action Foundation (NCAF), the national Community Action Partnership, the National Association of State Community Services Programs (NASCSP) and by the HHS Office of Community Services.

B.  THE SIX ROMA GOALS

    7. What are the national goals for use by CAA’s? 
   
There are six ROMA goals developed by the MATF and adopted by OCS for use by all CSBG “eligible entities,” most of which are CAA’s. They are:

The Six ROMA Goals

Goal 1.

Low-income people become more self-sufficient.

Goal 2.

The conditions under which low-income people live are improved.

Goal 3.

Low-income people own a stake in their community.

Goal 4.

Partnerships among supporters and providers of service to low-income people are achieved.

Goal 5.

Agencies increase their capacity to achieve results.

Goal 6.

Low-income people, especially vulnerable populations, achieve their potential by strengthening family and other supportive systems.

These are agency-wide goals.  Under these, each CAA lays out its specific goals for programs or strategies.

 8. CAA’s are all very different, so how can we all use the same goals? 
   
The six ROMA goals cover MOST – but not all - of the work done by CAA’s nationwide.  The design standard was to capture at least 90% of what CAA’s do nationwide.  Every CAA is unique and makes local decisions about programs and services based on local needs.  However, the MATF felt that a CAA must be able to report some activity under at least one of the six ROMA goals in order to be called a CAA.

 9. Are the six ROMA goals just for CSBG funded programs and services or for the entire agency and all of it’s services?
    In the Information Memorandum #49, the OCS states that the six ROMA goals should also be each state’s and each CAA’s goals. They recommend using the six goals to rethink and redefine the CSBG and CAA mission, to re-align services, to empower and re-energize staff and to evaluate effectiveness.  OCS states that such an approach is “both necessary and appropriate.”  Many people in the national CAA network believe that unless we do so, Congress will look less favorably on future CSBG funding. 

10. How can we organize all of our programs and services under the ROMA goals?
    You do not have to reorganize your agency’s operations to use ROMA.  Most of our programs and services fit under at least one of the ROMA goals.  Goal #1 covers most of the services we provide directly to low-income people to help them become more self-sufficient. Goals #2 and #3 address the work we do to help low-income people improve their communities and to become “stakeholders” in their communities.  Goals # 4 and #5 address the work that CAA’s and other eligible entities do to build community partnerships and strengthen their internal operations so that they can be more effective in ending poverty.  Goal #6 covers other human development programs (Head Start) and quality of life services (food distribution).

C.  OUTCOMES AND INDICATORS

    11. How do we plan for Results? 
   
Under ROMA, results are often called Outcomes”.  An outcome is the change that happens as a result of what we do. Instead of counting the number of services we provide (our “output”) ROMA asks us to measure the impact of the service we provide (the “outcome”).  At first, these terms may be confusing.  One way of looking at this change is -- if we can measure it by watching our staff as they work, it is probably a program “output”.   But, if we have to observe the service recipient, their family or the community, or ask them about it, then it is probably an “outcome”.  Outcomes are of course measured after the strategy or program has been operating.

12. What is an Indicator?  
    An Indicator is a measurement that helps you determine whether or not an outcome has been achieved, or whether or not you are making progress toward the outcome.  An Indicator is some show of evidence or proof that your program is working.  For example, one outcome measure could be “increase the assets of a family” and one indicator could be “increase in dollar value of savings account.”

13.  How do we go about designing the Outcomes we want?
    The development of outcomes, results and indicators is at the very heart of ROMA. This work usually requires that boards, staff, program participants and other stakeholders think through the agency or program goals, strategies and activities, agree on the realistic results that you hope to achieve (your “Outcomes” or “Results”) and the ways you will measure your progress (your “Indicators”).

14.  Where can I go for help in developing or measuring outcomes or results?
    There are now many successful ROMA projects around the nation.  The most successful examples are in those states (PA, WA, NY, IL, MO, MS,) where the CAA’s have decided to work together through their state CAA association and in collaboration with their state CSBG office, creating task forces that work collaboratively in developing measures that all CAA’s can use. For example, in Missouri the CAA’s produced a set of outcome measures for Goal #1 (the self-sufficiency goal) that all Missouri CAA’s now use.  This is a long process and it is not unusual for it to take 2 to 3 years to complete. 
    In addition most State Associations, including Cal-Neva, are building their training capacity by having people from CAA’s become certified ROMA trainers through the “Virtual Outcomes College,” a Pennsylvania group which is funded by the OCS.
    The National Association of State Community Services Programs (NASCSP) operates a clearinghouse of ideas on ROMA. See the ROMA website at www.roma1.org for almost everything in print about ROMA.

D.  REPORTING

15.  Is ROMA just a new way of reporting the services we provide?
    No.  ROMA is a new way of reporting on the results of the services we provide. ROMA requires us to report on the changes in individuals, families, and communities and/or changes in our agencies that have come about as a result of our work.  We are no longer just counting the number of services we provide.  We are also measuring the results we have achieved from those services. This is difficult work and often requires us to re-examine our strategies to ensure that we can obtain, measure and report on the results we seek to obtain.

16.  Does this mean we can no longer deliver or report on emergency services?
    No. CAA’s may continue to deliver and report on the emergency services they provide in their communities.  Some CAA’s integrate emergency services into self-sufficiency programs and family development efforts, and report those under Goal #1.  Generally, however, emergency services are reported under Goal #6.

17.  We find some of the data we have collected in the past to be very useful. Do we have to abandon the data collection we already do?
    No. ROMA is not the only information your organization may collect or find useful. Each organization should collect the data that is most useful to them in fulfilling their mission.

E.  APPROACHES to IMPLEMENTATION

 18.  What are the most common approaches to ROMA?
    There are three levels of approaches.   The largest amount of change is where you adopt (or adapt) an entire planning and management framework to re-design your CAA to focus on results.  If you are not doing agency-wide change, you can work at the program or strategy level, where you can use a “logic model” as a framework for revising each program to produce clearer and stronger outcomes. Or, within a program you can adopt or adapt specific tools such as “scales and ladders” and surveys that help you measure results and report on a program. Each of these approaches is further described below:

    A. Agency-wide management framework: two approaches 

a)  The Virtual Outcomes College is using the Drucker Foundation’s Self Assessment Tool in their ROMA Peer-to-Peer training course. The Virtual Outcomes College (with funding from OCS) has adapted the Drucker Self Assessment Tool for use by CAA’s.  It is a comprehensive framework of vision setting, mission review, goal setting and reporting. 

b)  The Rensselearville Institute in New York has developed a planning and management system for use by CAA’s that has been adopted by CAA’s in about a dozen states. The state or national CAA Association can provide contact information on both of these systems.

Using Logic Models for a Program, Strategy or Goal.
    A logic model lays out the purpose, goals, activities, results desired, and measurements for an entire strategy or program. Typically these are laid out on one page in adjacent columns.  It is a useful way to visually see the relationships between activities and outcomes.  The logic model, developed to improve the ability to evaluate the results of program, went into widespread use in human services in the 1980’s. The OCS Demonstration Partnership Program required development of a logic model for each of the 100 projects they funded. Examples of logic models for family development, minority male, micro-business and other programs are included in the evaluation reports on those projects that are available from OCS.   The Early Head Start Program also requires development of a logic model. 

Specific Tools: Scales, Ladders and Surveys

    Scales and ladders are a way of measuring incremental change in individuals, families, communities or agencies. One familiar ROMA scale and ladder system currently in use measures a participant/client, agency or community as they progress from “In Crisis”, to “At Risk”, then “Stable” and then, hopefully, to “Safe” and then to “Thriving”.  For example, a family may be “in crisis” because they are homeless.  They get a temporary job and move into an over-crowded or unsafe apartment, so they are no longer homeless but still “at risk.”  Over time, with assistance from the CAA, their income increases and they obtain a more appropriate apartment - so they are now “stable.” As the family moves from up the scale or up the ladder from “in crisis” to “at risk” to “stable,” the CAA records the families progress and the CAA’s efforts to help the family make this progress - and reports each step under a ROMA goal. Another useful measuring tool is a survey, a series of questions that provides feedback from a group of respondents. Like the customer satisfaction surveys used in restaurants or hotels, the survey ask a series of questions about what happened to you and how you feel about it. The results of the survey provide useful management information.

F.  MEASURING YOUR PROGRESS TOWARD ROMA IMPLEMENTATION

19. How can we assess our progress and ensure we are moving in the right direction toward full use of ROMA?
    The change to ROMA will take most agencies several years to complete. There are four basic ways to measure your progress.  They include: 

(a) Comparing where you were last year with where you are today (Are we making progress?).

(b) Comparing your methods and progress with those of other CAA’s (benchmarking with our peers).

(c) Talking with people who have ideas about results measurement, and

(d) Comparing your progress with the national expectations and norms as described by OCS in their Information Memorandum #49, dated February 21, 2001 Responsibilities and Strategies – FY 2001-2003.  On page six, OCS describes expectations for all entities eligible for CSBG funds with regard to ROMA as follows:

1. The entity and its board complete regular assessments of the entity’s overall mission, desired impact/s and program structure, taking into account: 1) the needs of the community and its residents; 2) the relationship, or context of the activities supported by the entity to other anti-poverty, community developments services in the community; 3) the extent to which the entity’s activities contribute to the accomplishment of one or more of the six ROMA national goals;

2.  Based upon the periodic assessments described above, the entity and its board has identified yearly (or multi-annually) specific improvements, or results, it plans to help achieve in the lives of individuals, families, and/or the community as a whole;

3.  The entity organizes and operates all its programs, services and activities toward accomplishing these improvements, or outcomes, including linking with other agencies in the community when services beyond the scope of the entity are required.  All staff are helped by the entity to understand the direct or indirect relationship of their efforts to achieving specific client or community outcomes; and

4.  The entity provides reports to the State that describe client and community outcomes and that capture the contribution of all entity programs, services, and activities to the achievement of the outcomes.


ROMA:  Where it came from and why are we doing it?

ROMA stands for Results Oriented Management and Accountability.  Remember management-by-objectives?  This is managing-for-results.  In 1993 Congress passed the Government Performance and Results Act.  All Federal agencies and funded programs must use the GPRA system.  ROMA was development by a Task Force from the CAA network to implement GPRA in the CSBG network.  ROMA is the mandated system that all CAA’s nationwide must use.

    So the big reason we have ROMA is because of GPRA, and the big reason we have GPRA is because of a fundamental change in social values by the American public.  In the 1970's, the level of confidence the American people and Congress had in government in general and in publicly funded programs began to decline.  Some say this was caused by the Vietnam War.  Other say it was Watergate and President Nixon’s resignation.  This manifested itself in the 1980's in the form of President Reagan saying in his Inaugural Address that “Government is the problem”  The newspapers ran stories with allegations of “waste, fraud and abuse.”

    In 1990, Congress Amended the Chief Financial Officers Act to require better explanations of expenditures.  Congress quickly realized that this kind of reporting was only the caboose of a very long train, and they needed to describe where the train was going, the engine and the other cars.  So they decided to require each and every Federal agency to develop a strategic plan, with a vision, goals, and measures of the results and outcomes being produced.

So Congress passed, WITH NO DISSENTING VOTES the Government Performance and Results act of 1993.  It requires every Federal agency to prepare a strategic plan that includes a vision, a statement of values, long-term goals and results and outcomes. 

For the past 50 years, almost all requirements for planning, management and reporting placed on any Federal agency wind up getting applied to the programs they fund.  This means that the framework being used by Federal agencies for strategic planning and identifying outcomes is trickling down to the funded programs.

So all agencies began implementing GPRA.  In HHS, there is one strategic plan and eleven sub-plans.  One plan is for the Agency for Children and Families (ACF), where Head Start, LIHEAP, and CSBG are located.  It is about 200 pages long.  You can download your very own copy fromwww.acf.dhhs.gov/programs/opre

     On the right side of this home page there is an entry that includes “Planning -- annual performance plans and reports “  Click on it.  Then on the next page click on “FY 2004 performance plan and FY 2002 performance report.”  It is in sections, with the major sections being the goals.  In theory you can download either Word files or pdf files, but I have not had much luck getting the Word files so here is the pdf file location. For example, Strategic Goal 3 is where CSBG is located (starting on p. 125) is at www.acf.dhhs.gov/programs/opre/fy2004performance/fy2004finalplan-section4.pdf

 Strategic Goal # 2 includes Head Start. It is at 
http://www.acf.dhhs.gov/programs/opre/fy2004performance/fy2004finalplan-section3.pdf

But let’s go back to the history of the development of ROMA.  A national Monitoring and Assessment Task Force (MATF) was created in 1994 to apply GPRA to the Community Services Block Grant.  The MATF has about 75 people on it from all the major associations created by CAA’s and State CSBG agencies, including NCAF,  NACAA, NASCSP, OCS and others. They developed ROMA to implement GPRA in the CSBG network.  The developed six goals and many suggested measures.  (See Appendix for the goals and measures.)

     The top management of the United Way of American and H.H.S. typically work on the same issues.  They go to the same conferences, socialize with each other, and work together on most human development issues.  In the mid 1990's, The United Way of America created their own task forces and training materials to help the networks of voluntary agencies (American Cancer Society, Goodwill Industries, Boy Scouts, The Urban League, and hundreds of other voluntary agencies) to improve their results and outcome measures.  See http://national.unitedway.org/outcomes/publctns.htm

     The reality, then, is that the shift to demand more accountability for outcomes is about 20 years old and applies to all Federal agencies and as being done by virtually all other networks of nonprofits.  The issue is not whether or not we have to do this – the only issue is how to do this.

    Too many explanations have been put forth to justify ROMA that rely on the “why this is good for you” type of argument.  While we hope there are some specific benefits, it does not really make any difference whether fund raising is made easier or more difficult; or whether it makes reporting easier or not, or whether it makes planning or staff accountability easier or not.  Efforts to sell ROMA at this level of argument meet resistance from people for whom the status quo is good enough, or who do not see why the effort it takes to switch to a new approach is worth it in terms of payoff.  All these arguments and their adherents miss the big issue.  There was a nationwide shift in social values among the public and in policy by the U.S. Congress and other funders (including the United Way). They want to know what results the expenditures are producing in the individual, families and communities and EVERY PROGRAM is being required and challenged to do this.   Nobody has “escaped” from the need to do this.  This is a do-it-or-die situation.  The only question is how much time any given program has to get this done before the axe drops.

    Within a few years all Federal agencies will be submitting their budget requests to Congress based on the results the proposed appropriations will buy.  Six states are now switching to a results-based budgeting, including Iowa and Washington.

    So then we move into the how.  Within the community action network, attitudes nationwide vary across programs, agencies and individuals – from head-in-the sand minimalists to cheerleaders shouting from the pulpit of change.


2.  Don’t get overwhelmed with ROMA.
    Being a committee-created document, ROMA incorporates a range of ideas and a number of compromises.  Before it was unveiled, one group of advocates were presenting ROMA as a theory and strategy of family development to produce self sufficiency, to be measured with scales and ladders.  Another group has integrated it into a theory of organizational change requiring a “complete transformation” of what a CAA is and does.  Another group argued that everything was just fine, and all we had to do was add a couple of additional reporting categories.

    The history is that CAA’s have had requirements to produce strategic plans that describe their results and outcomes before. The first was in the late 1960's when the Office of Economic Opportunity asked every CAA to create a strategic plan, and to report on their results using about 150 nationally standardized program accounts.  (Example: PA 132 adult basic education, was to raise literacy to the 8th grade level  PA 133, GED Prep, was to help participants obtain a GED.)  The second was in the 1978-1980 period, when CAA’s were required to produce a multi-year plan that included a description of the changes they would produce in the families and the communities they worked with.  These requirements of that new Grantee Program Management System (GPMS) were fully implemented in Regions I to VIII.  CSA had spent about 15 million dollars and over 100 trainers had trained about 8,000 people in this new system.  However, the GPMS was barely introduced in Regional IX and X when the block grant passed in 1981 and the Federal Community Services Administration (the predecessor to the Office of Community Services) was disbanded.  Several states picked up the  GPMS approach and continued with some or all of the elements of the system.

3.  How Big is ROMA?
    If you took a person from another culture to the mall in December and walked out into the atrium and asked them to describe a Christmas Tree, they would say something like this:

 "It’s about 50 feet tall, and has huge silver and gold balls on it, with candy canes about two feet long, hundreds of multi-color flashing lights, and lots of white flocking on the branches.”  And on and on.

    The definition we really want would be something like this:

 “An evergreen tree, any of 4 or 5 species.  Usually about 6-8 feet tall.  Usually has lights and other decorations.”

    So this tool kit is going to take us back to a simpler, more straightforward explanation of the core elements of ROMA.  In this process of de-constructing ROMA from the ideas it has accumulated, lets start with the management framework in which ROMA operates.


4.  ROMA and Management System.
    ROMA can be used with any management system.  Some people have presented it as requiring use of one particular management system, or at least requiring major changes in your management system – whatever it is.

    Since many people do not know what their management system is, or have internalized it so that it is indistinguishable from their personality, talk like this creates a lot of anxiety.  Let’s take a step back and look at the three major types of management systems:

         1. Command and control
    2. Systems based
    3. Team based/quality improvement focused

The classic hierarchical command and system has been in use for about 500 years.  It was developed in the middle ages and used in universities, armies, and churches.  The educated few at the top (the brains) issued the orders and the lower levels (the hands) carried them out.  This was implemented in the Industrial Revolution with the “Taylor” style of breaking tasks down in to smaller and smaller pieces.  Most government agencies and public programs still operate this way.  By the1950's Ford Motor Company had 15 layers between the top and the bottom. 

With rising education levels and almost all workers now being literate the managers realized that with every pair of hands you get a free brain.  With computers to move data, many of the middle layers became irrelevant.  In the 1980's and 1990's most U.S. Corporations “de-layered” their structures.  Ford Motor Company now has 5 layers between the top and the assembly line.  The average number of workers-per-supervisor in U.S. businesses has increased from 6 to 20.

Looking at the command and control charts, in the 1960's the University business school professors and consultants (McKinsey and Company) concluded that the “white space” between the boxes represented an absence of authority and led to slow processing and incomplete hand-offs of work. They invented the business systems model in which all the wheels-and-gears were interlocked, like a watch.  This theory is still the framework in which almost all Graduate Schools of Business organize their curriculums and issue MBA’s.  In the 1980's and 1990's, the NACAA used this framework for their CAA Executive Director Manual and other manuals.  In 1998, the Head Start program adopted this theory in their PRISM (Program Review Instrument for Systems Monitoring).

In the 1980s, the quality guru’s (Deming, Juran, Crosby, Peters, Wheatly, et. al.) began to point out how the bureaucracy was limiting innovation and creativity and inhibiting responsiveness to customers.  They came up with the idea of “flat” authority structures in which self-managing teams would clarify their sense of mission and values, and then take on a function and organize themselves to complete it.  Many businesses and a few nonprofits have adopted these theories.

While the type of management system and the skills of the managers certainly affect employee morale, productivity and customer focus, all management systems face the challenge of describing how the program strategies they have selected attack the causes of the problems and produce changes in individuals, families and communities.

ROMA can be made to work in any of these types of management systems.  I believe it is easier to figure out how to implement ROMA if it is un-enmeshed from the management system.


5.  ROMA  does  require a change in thinking about the weighting of the elements of the program model
.
    The classic model, in use since the 1930's, is inputs, activities outputs, and outcomes.  Inputs are things like money and people.  Outputs are the tallies of how many people did what, e.g. attended classes, read books.  Outcomes are what happens outside the staff- to program participants, their families or the community.  (In marketing terms outcomes are the benefits produced.)
     Traditional emphasis on the amount of narrative and reporting attached to their categories might look something like this:

  Inputs Activities Outputs Outcomes
Traditional/past 40 10 40 10
What GPRA wants 10 10 10 70
What Purists want (tell us what we bought) 10 0 0 90
Where we are in transition 30 10 40 20

Put another way, the purists would have the Federal agencies go to a Zero Based Budgeting System (used by President Carter).  “Assume your budget is zero.  Tell us what results you would produce and how much money these results will cost.”  This has recently been re-emphasized in the Bush Administration as part of the President’s Management Agenda (see the OMB web site at http://www.whitehouse.gov/omb/budintegration/pma_index.html

     This lays out his ideas for changing Federal agencies and enhancing outcomes measurement.

    We are in a transition phase, and many agencies are having a difficult time in improving their results and outcomes.  One reason for this is that instead of changing the program model and changing the assumptions, strategies and work program descriptions, most funders have instead added a new template on top of existing requirements.  Most funders have left program formats, strategies, and budgets in place, and added a new reporting form.  The approach of leaving all existing elements in place and adding a new section on outcomes may make people believe that this is “just another reporting” requirement.  This may reduce the urgency of developing the results and outcome measures, or of changing strategies to produce better outcomes.

Given the enormous complexity of coming up with results measures for most human development programs, this is understandable.  The task is further complicated by the current status of social science, which does not offer immediate solutions to our dilemma (See below).

So if we have boiled ROMA down to its essential element – an increased focus on outcomes.  In the process of reducing ROMA to its core elements we have also removed any bias toward a specific type of outcome or a specific intervention strategy is removed or a specific method of measurement.  What, then, is left in the pot?

What is left is the urgent need to develop new and better ways of describing the changes produced in individuals, families, communities as a result of our work, and to develop better ways (strategies) of producing desired changes.  You might think that having de-constructed this Christmas tree and boiled off the decorations that what is left would be simple to deal with.  Unfortunately this is not the case.  Describing outcomes is a huge challenge.  There are several reasons for this.

  1. Humans tend to confuse ideology (motives or good intentions) with results.  This is a classic in human services – people want to be judged for their good intentions not what changes they produce.

  2. Most Federal statutes are a product of compromise. They have numerous internal contradictions built into them.   (Cite GAO reports 1986 and 1987).  The muddled legislation represents compromises and messy combinations of theories, causes, strategies, programs, results, activities, participant groups, etc. You can NOT get from most Federal statutes to a coherent group of results unless you do a lot of “social construction” in the middle.  Logic models can help us address these problems by creating a framework of goals, activities results and measures.

  3. Until the 1980's, the assumption was that you described your program in the grant application and then ran it that way for the whole year.  This enables funders and evaluators to see which of your activities worked and which did not.  Starting with the quality movement that empowered from-line staff to make changes every day to meet customer needs, the static program model began to erode.  CQI, TQM and other approaches now legitimize constant changes and learning as time passes.  This makes it difficult to measure efficiency or cost-effectiveness.

  4. Results measurement requires great clarity about “what is to be done” but – in many agencies goals are often vague and open ended.  Even if the goal is clear, we usually do not know what is to be done in any specific situation until there is a (private) interaction between the staff person and the program participant and they reach an agreement about what the participant will do and what the staff will do.  So the  “what” varies significantly from participant to participant, and is unknown until the staff and that person reach an agreement.

  5. Much of what we seek is to promote a social movement for social justice reasons (greater equity, fairness) which are difficult concepts to measure, and take decades to accomplish.
        Now we have identified some of the small problems, let’s look at the BIG problem. 

  6. Limits of Social Science.  Most sociology, cultural anthropology and political science theory is big theory that applies to large groups of people. It is not a small theory that is true for every individual.  You can not apply big theory to individuals.

     Education has been developing measurement tools for 100 years; sociology for 75 years, and psychology has been developing tests for 50 years.  These measurement tools are developed incrementally and get better as time – usually decades – passes.  Many of these measurement systems or tests have been “normed” on millions of people.  Most of them have to be administered by a trained person.  And, they still have rigid limits.  Most are univariate – they measure one trait.  If you try to use the test for anything outside that single trait, it does not work.  We read stories weekly about the controversies of using tests in public schools.

If you have a program where you can use hard science like physics and chemistry and engineering and micro-economics to measure results, like WX., you can do a good evaluation. The Oak Ridge evaluation of WX is an excellent example.  Unfortunately, most human development programs are not based on these sciences.

So how DO we create results measures?  This is a process of social construction. We must create a miniature theory that includes:

* what the condition is
   
* What the cause is
   
* What strategies will change the cause and thereby change the condition
   
*

This is an iterative process; we try something and then improve it.  It takes years to develop good outcome measures.  Also, just to be clear, I think that scales and ladders ARE a good tool for CAA’s, because they enable you to measure incremental change over time.

This is a challenging process.  If every local program has a slightly different condition or strategy, how do we compare them?  If we standardize measures on a nationwide basis, then how to we respond to unique conditions?  There is a major debate taking place at the national level about how we should add up the work that all CAA’s nationwide are doing.  Under the current system, CAA’s select their own goals and outcomes.  The Bush Administration has proposed that a dozen or so measures be used for all CAA’s, and that each CAA will assume a proportional responsibility for producing on that measure.

So, the development of results and outcome measures locally requires a LOT of conversation, analysis and thinking about why people are poor, what we can do about it, and how we measure it.  The Community action network is now nine years into this process (remember we started in 1994) and making good progress.  Every other Federally funded program is going through the same process.  We are ahead of most of the rest.  Keep up the good work! 

The next section will provide ideas on ways to develop results and outcome measures at the local level. 

G. ROMA.  What is an outcome and how do I develop one?

Outcomes and results:

     * help to explain and illustrate the multi-year goals.

    * a

    *

Generally, outcomes can:

    * be measured.

    * u

    * describe the desired condition/s that will exist and how that is different from the existing condition/s.  

 
One simple method for developing expected outcomes is to complete the sentence: 

“I will be able to show that this goal has been achieved when . . .”

Generally, to identify an outcome you have to look outside the agency to the program participants and the community.  If you can measure it by watching the staff, it is probably an output.

1. Conceptual approaches to developing of the CONTENT of outcome measures.

What are the types of outcomes?

          Because of the way G.P.R.A. and ROMA and Congressional mandated outcomes are coming at you, the selection of specific measures to be used for reporting at the local level or within a state is often presented as the starting point for discussion.  At first glance, that appears to be the challenge.  This probably makes the implementation process more difficult, because the discussion assumes the rest of existing program operations are a given and that only one thing – reporting measures – must be developed or changed.  Wrong and wrong again. 

            Whether you are looking at an agency or a program, changes in any one part of the system are going to precipitate changes in other parts of the system.  If we change the measures, then everything above them (e.g. goals, objectives, strategies, activities) is going to be affected as well.  The outcome measures selected become both crucial descriptor of but also drivers of program strategy, because in order to produce the desired result you have to use a specific strategy that affects that measure.  Put another way, Julie Jakopic -- formerly of NASCSP -- says that “Because we end up doing what we measure, we need to choose carefully how we measure what we do.” 

           So we must consciously broaden the discussion about implementation of the new reporting system to also look at the conditions in the community, to include our theories about why the society works the way it works, and to include a review of the strategies we use to change it.   I know that this expansion of awareness takes time and many people resist doing it, because helped I peel this onion over about a three year period with USDA rural development programs.  Any expansion of discussion to this broad range of topics immediately provokes resistance, especially from the defenders of the status quo.  To them, an effort at large scale change implies that what we have been doing is somehow wrong, that we are guilty of not doing what we were supposed to be doing, and so on.  One element of change management is to ELIMINATE RESIDUAL GUILT about changing something.  This is done by the leaders who say things like:

“What we did was not wrong, it just was what we were doing.  We did the best we knew how to do at the time, now we are looking for something else to try.  Don’t feel bad about the past – feel good that we learned from it.  Now, it is time to move on. 

      There are several ways to go about developing new measures of results and outcomes.  You may use more than one approach, but it is useful to unravel them and look at the different assumptions on which they are based.  And, no matter which way you start, you are probably going to wind up moving to a review of most of your program strategies.

            A.  Inductive approach from existing program operations.  Start with what you’ve got, usually program outputs, and see how far you can stretch toward a description of outcomes.  In working with the U.S.D.A. Rural Development mission areas, John Johnston and I assisted several working groups (housing, business, utilities) in development of results measures.  We found that we could construct usable outcome measures (1) by starting with the programs existing activities, milestones and output measure(s), and (2) working outward very slowly and carefully toward the family and community, and (3) tracing every step, (4) to make sure there was a powerful link between every step.  This was an inductive strategy -- to build what the evaluators call a logic model -- that starts with the program and moves outward by inches.  And, it inevitably takes you into questions about basic program strategy.

            B.  Deductive approach from a plan.  Start with community conditions and a vision mission and goals.  This is the approach used by The Rensselearville Institute.  Construct a far-reaching plan, identify your  goals, and then figure out how to determine if the goal is achieved.  Ask yourself: “How would I know if this goal had been achieved?  How would I measure it?”

            C.  Use a deductive approach that starts with social indicators and tries to bring it down to connect with a program.  In recent years, approaches such as the Oregon Compact have used social indicators (the Oregon Benchmarks) as a way to focus attention on a subject area.  A large-scale citizen planning effort led to the selection of the 259 measures, which were then boiled down into a set of about 20 high-priority measures.  They are used to focus attention on a topic, e.g. “Let’s all do what we can to reduce teen pregnancy.”  But they are not used as a way of measuring performance of individual programs – they are used as signals that this is an important topic and people should be working on the issue.  This also seems to be the approach being used by some of the Empowerment Zones and Enterprise Communities (EZ/EC initiative), but some EZ/EC’s appear to have confounded both social indicators and benchmarks.  It is hard to figure out how changes in the small populations covered by the zones are going to show up in social indicators.

            Remember, in the 1970's both the United Way and the H.H.S. (then HEW) made a serious effort to use social indicators to measure progress in the social programs they funded – and they abandoned it.  It did not work because most social indicators are the consequence of dozens or hundreds of factors that are working together to produce that result.   The unemployment rate, for example, is impacted by interest rates in Thailand, high-school graduation rates in Japan, migration patterns in Mexico, technology patents in Germany, and on and on.  Conclusion.  DO NOT adopt social indicators (unemployment rate, crime rate) as outcome measures for any of your efforts. 

            The H.H.S./O.C.S. Monitoring and Assessment Task Force (M.A.T.F.) is suggesting you use social indicators only as descriptors of the CONTEXT of your work, i.e., the unemployment rate in the community may influence your ability to place people in jobs, but you should not assume responsibility for tying to change the unemployment rate per-se.

            D.  Customer satisfaction measurement.  You can make a pretty good argument that our focus on program outputs is mis-focused because it gives too much credence to what we as program managers think the service “is”, and that in a service business the only thing that really matters is what the customer thinks has happened.  Most service businesses  –airlines, hotels, restaurants, banks, doctors, accountants – ask their customers what they think and give them ways to give them systematic feedback through mail-in or drop-in-the-box survey forms.  Most Federal agencies now do the same thing, including the Social Security Administration and the DOL employment and training programs.  Most cities and school districts also measure customer satisfaction.   (The author is highly opinionated on this subject and in the interest of full disclosure should report that he develops and conducts customer satisfaction surveys for Head Start programs.)                   

            E.  Expert Observations.   Use professional opinion of the school counselor, teacher, psychologist, or social worker. You are in effect relying on the credibility and authority of this expert to validate the measurement.  One Head Start Director in Virginia asks all the school counselors “Do you see any difference between the Head Start children and the others.”  And they say “Yes.” And he says “Would you write me a letter on your letterhead stating that.”  And they do, and so he has a stack of letters validating the program that he carries in his briefcase to show to people.  Clever!

            F.  Combine Customer Observation with Expert Observation.  Develop scales and ladders to connect what you do with the results produced.  One of the useful tools adapted for use in ROMA are scales and ladders. 

            These rely on the combined judgment of the participant and the staff person (the expert).  They are a good tool to use to measure incremental change over time.  There are several examples on the ROMA web site, at www.roma1.org

            You can use the scale and ladder approach to measure just about anything.  Put the worst case scenario at the bottom, the best case at the top.  Create intermediate steps that show progress from the worst to the best.  Eureka – a scale and ladder!

2.  There are many SOURCES you can use to identify existing outcome measures

            A.  For Head Start.  Select measures of outcomes and results from IM’s, and the many research studies.  For IM’s and other Head Start publications, go to the Head Start publications center at www.headstartinfo.org/searc.htm

            IM 03. Initial Guidance on New Legislative Provisions on Performance Standards, Performance Measures, Program Self-Assessment and Program Monitoring. Issued in 2000.  Copy Appendix A and B at  http://www.headstartinfo.org/publications/im00/im00_03.htm

             IM 18.  Child Outcomes, Performance Measures, Program Self-Assessment: Using Child Outcomes in Program Self-Assessment   Be sure to also copy Appendix A which has the Child Outcomes Framework.  http://www.headstartinfo.org/publications/im00/im00_18.htm

             They frequently change the specific links given below, but you can always keep track by starting at the ACF homepage at http://www.acf.dhhs.gov/ then go to ‘Research and Publications’, then go to ‘Head Start Bureau Research’ then to ‘Ongoing Research’ For example, FACES is at www.acf.dhhs.gov/programs/core/ongoing_research/faces/faces_intro.html

             The Head Start Quality Research Consortium has several reports on program practices and outcomes. Check it out at: www.acf.dhhs.gov/programs/core/ongoing_research/qrc/qrc.htm

            The site for the Impact Evaluation of Head Start is at
www.acf.hhs.gov/programs/core/ongoing_research/hs/impact_intro.html

            and the impact evaluation for Early Head start is at:                    www.acf.dhhs.gov/programs/core/ongoing_research/ehs/ehs_intro.html

            The Head Start/Public Schools Early Childhood Transition Demonstration program reports are at www.acf.dhhs.gov/programs/core/pubs_reports/hs/transition_intro,html

            For Head Start and Preschool, use the County of San Bernardino County, Human Service System, Preschool Services Department Measures.  (909.387.2375)  The Florida R&D projects have new measures.        

            The California Head Start Association and the State Dept of Education negotiated a mutual set of school readiness measures and other outcomes, called the D.R.D.P.+    Both are seeking the same desired results.

            And, use logic models.

 

H.  For Community Development

     Most of the Community Development Corporations, including those funded from the O.C.S. discretionary funds, are involved in the Success Measures Project funded by the McAuley Institute and operated by the Development Leadership Network, Baltimore, MD.  They created working groups in ten locations around the country to try to create outcome measures for the complex community-development and community- building projects.  They struggled with all the same issues address at USDA, HUD and H.H.S. efforts, but did not use much of the experience from the other agency efforts.  It was more of a “whole cloth” re-invent the wheel type of effort, which of course can sometimes produce fresh new tools.  The topics areas in which they worked are: Housing, Business and Job Development, community participation, levels of hardship, and the relationship between funders and implementors.  Their initial publication is “Success Measures Guide Book version 1.0”. The Development Leadership Network, 685 Centre Street, Boston, MA 02130, 617-971-9443.  Their FAX 617.971.0778.   Contact info@developmentleadership.net, and their guidebook is on their web site at:  www.developmentleadership.net

            In the meantime, a very useful tool is already available from the Virginia CAA’s.  They developed and piloted this in five agencies.  It is the “Community Matrix Focusing on Neighborhood Organization and Civic Capital.”  It is a type of scale-and-ladder that shows how you can help a community move from dis-organization to higher levels of functioning.  This is VERY useful – it is a major contribution to our field.  Thanks to the Virginia Head Start Sponsors for developing this, and thanks to Ted Edlich and others at Total Action Against Poverty for making it available to others.  WRITE to:  Ted Edlich; TAAP, 145 Campbell Avenue, Roanoke, VA 24011.  The Illinois CAA Association and Missouri Associations are also working on these measures.  Pennsylvania has some as well.  Ask your staff to collect some of these and look them over.

            The North Central Regional Central for Rural Development, Iowa State University, also has a useful publication Working Toward Community Goals: Helping Communities Succeed that combines both community development strategy and reporting of results.  It came out of collaboration between the Aspen Institute, Ford Foundation and USDA.  Their web page is: http://www.ag.iastate.edu/centers/rdev/RuralDev.html

            Also, check out the material produced by the Northern California Council for the Community at http://www.ncccsf.org/    Their publications are Goals and Success Indicators of Healthy  and Self-Sufficient Communities, and  is Bay Area Partnership: Building Health and Self-Sufficient Community for Economic Prosperity, Status Report.  This is based on use of social indicators and is similar to Oregon Benchmarks, but brought down to a community level. 

            A VERY useful publication to use to look at all the Federal block programs in terms of their purpose, structure, relationship to states is the GAO publication, Grant Programs: Design Features, Shape, Flexibility, Accountability and Performance Information. 

            Our whole universe is in motion!  All Federal agencies are changing their program and reporting structures.   You are being asked to “coordinate” and “plan with partners” but Federally funded program structures are changing and will be with us for several more years.                       

            C.  The United Way as an excellent source of info on how to measure human service programs.  (Tab 18)             http://national.unitedway.org/outcomes/publctns.htm

            D.  The bibliography in this publication lists the International City Management Association and other good sources of methods for developing results and outcome measures. 

            E.  Mix and Match.  Most agencies use a mixture of the approaches described above.  It is important to be clear about which method you are using, otherwise you may find yourself where you think the discussion is about the content of a measure when it is actually about the logic system you are using to develop the measure.  

3.  Group process approaches to development of results and outcome measures.

            There is an argument to be made that the process you create for this change effort is much more important than the content you create, because the process will be used to modify, evolve, revise and change the content for the next several years.  There is no such thing as developing results measures and using them forever.  Next, we provide four examples to illustrate some of the possibilities in developing outcome measures.

1. Every Program Does Its Own Thing.    The funders take a hands-off approach and let each program develop whatever it wants.

Advantages.

    * M

    * P

   Disadvantages. 

    * Unable to relate each local program’s activity to the activity of others.

    *

    * Slow – everybody has to reinvent their wheel.

    *

    *

2.  A Federal or State Agency Mandates the System.

   Advantages.

    * F

    *

    * M

   Disadvantages.

    * C

    *

    *

    * Resistance to “Federal or state bureaucracy.”

3.  It Is A Group Effort, But Factions Form among the local programs, and one group is allowed to win.   One faction consists of the “lumpers” who want to combine a lot of types of activity into very general measures, the other consists of the “splitters” who want to develop detailed and very specific measures.  (These types of factions also form among linguists, and among ethnologists who study race.)  The two factions reach a stalemate.  One faction outlasts the other and “wins” and imposes their definitions on all programs.  The losers have a serious case of sour grapes and evade use of the new system as much as possible.