By Jim Masters
Center for Community Futures
January 2000
TABLE OF CONTENTS
Forward
A. Key concepts in ROMA
B. Where did ROMA come
from and why should we do it?10
DOL, USDA, HUD, HAS
D. Where do I get help locally? Performance
and outcome measurement in city and county government and at the United Way
...............................15
E. Can I get help from my local
college? Outcome and performance
measurement and the social sciences..............................16
F. How ROMA is the same or different
from previous program structures and reporting systems used by CAAs?
G. How ‘wide’ should ROMA be? Across which programs in a CAA?H. Who should lead the effort in
our CAA?30
I. PROCESS approaches to developing
outcome measures
Case Study 1 - Pennsylvania.................................33
Case Study 2 - New York ....................................34
Case Study 3 - Washington ..................................35
Case Study 4 - Missouri ....................................36
Case Study 5 - California ..................................38
Case Study 6 - Rensselearville..............................39
J. Bring your funders along…42
K. Possible roles for state CAA
associations43
L. Approaches to developing of the
CONTENT of outcome measures44
M If scales and ladders are as good as
this author says they are, why are they controversial in some CAAs?
N. Peeling the onion: updating the assumptions on which your current operation is based1
O. Peel another layer: review your purpose, vision & values
P. More content traps to avoid in system design
Q. Transition from the old to the new
R. Reporting system development
S. Linking your new
outcomes measures to your financial system
T. Software and ROMA
U. Future development
possibilities
R.O.I., . Community Services Network, Too much generic activity,
Persist!
Appendix A. The big picture
Appendix B. Where to put strategies under goals Forward65
Appendix C. Toward A Certified
Statement of Outcomes, by Frank Monti, C.P.A., Innovating Magazine
Bibliography
22 Steps to ROMA
Implementation: Peeling the Onion
Forward.
This workbook helps CAAs, State CAA Associations and State CSBG offices implement the Results Oriented Management and Accountability (ROMA) framework and principles as adopted by the CSBG Monitoring and Assessment Task Force (MATF). The assumption is that the reader is familiar with ROMA basics and has already read the ROMA Guide and the other materials published about ROMA, all of which can be downloaded from the ROMA web site at This workbook includes only brief references to ROMA as a context for the suggestions herein.
This workbook is primarily for people who have read the ROMA Implementation Guide and are still having some problems implementing ROMA, or who started to implement it and have stalled. The workbook provides:
* a decision making framework, describing some of the major choices you can make about implementation,
* the context for ROMA -- other approaches to outcome measurement in Federal, state, local governments and in other networks (“we are not alone”),
* previous management and reporting systems used by CAAs and implications for ROMA implementation (“front-line staff must love their reporting system in order for it to work”),
* how the six goals evolved and why their use is crucial to continuing the concept of the CAA as an umbrella agency,
* process options to use inside your CAA, at the state level and at the national level,
* criteria to use to develop the content of results measures (and traps to avoid),
* why some standardized results measures (and some optional) are needed at all three levels,
* why the controversy some people have with scales and ladders are really about the family development strategy,
* ROMA and your financial system,
* why creation of ROMA results measures inevitably leads you upstream into a review of your goals, objectives, strategies and programs activities, and
* on each topic, the author’s conclusions and recommendations.
Each topic has three sections. Background information is provided; the author tries to show a range of existing practice so you can “find yourself.” If you are in the right place for you and your agency then you stay there. If you should move up or down on the spectrum then do it. Next, the author states his conclusions (underlined.) Then, thirdly, the author gives his recommendations (bold face) for the steps you should take on that topic.
How to use this book.
You can skip topics and steps, or change the sequence. All materials cited are also listed in the Bibliography.
There are seven points about ROMA to be made up front.
1. This shift is real. Whether or not you think it is a good idea, the new emphasis on describing what you do in terms of results and outcomes is here to stay. It is driven by changes in social values, by the general public’s heightened expectations for effectiveness in government and by far-reaching Federal laws that passed with huge majorities in Congress. It is taking place in every agency and program of the Federal government, and it is taking place at the national, state and local levels in all publicly funded programs. This is not going to go away. This is something every CAA should be doing – and if you are not implementing it already, then get moving!
2. This is big. The shift involves more than just installing a new reporting system. It requires re-thinking what you are trying to accomplish with the families and communities with whom you work, which requires a re-examination the causes of their current situation, which then carries you into rethinking the types of activity in which you engage. So outcomes thinking will eventually take you into an A-to-Z review you your existing agency operations.
3. You can do this. This is the sixth major shift in management frameworks for community action agencies since the Economic Opportunity Act was passed in 1964. This workbook includes a brief overview of each of the previous systems (program structures and reporting and how ROMA is similar or different). You did the first five. You can do six.
4. This takes time. Two to three years is a typical time frame for an agency or a group of agencies to do the kind of collective learning process that takes them beyond the existing way of thinking. It is going to take five years to flesh this out at the local, state and federal levels. This five-year time frame is about the same as the last three shifts in management systems in the community services network.
5. There is good news. The Federal government and most state governments are not telling you exactly how to do this. The approach that OCS, NASCSP and NACAA have taken in leading this effort is to give YOU a lot of latitude in designing your approach. The “bad news” is that since nobody is telling you exactly how to implement ROMA, you have to make up a lot of this on your own.
6. The flexibility in implementing ROMA is good, but it creates a serious risk of diffusion if everybody invents their own measures and definitions. We must create a common language that we can use to communicate with each other and to retain the statewide and national character of our anti-poverty work together. Some approaches that address this issue are included in this workbook.
7. A key challenge is whether you use this new approach and the implementation effort to change your strategies, or just pour old wine into new bottles. In creating the existing reporting system in 1981 -- 1985, the intention then was to attach as much CAA activity as possible to the new CSBG. Then, we were trying to put old wine into new bottles. Today, in implementing ROMA, that would be the wrong choice.
Some people say that ROMA should be used only to tell the existing story better. I think there are some problems with the existing story. We have too many strategies that are based on obsolete notions of the economy or in muddled theories of human development. Peter Drucker advises us that about 1/3 of all goods and services become obsolete each year, either because of advances in knowledge, technology or competition. ROMA offers an opportunity to assess what is working and to identify what is not working too well. Alvin Toeffler, author of Future Shock, says that for the next millennium we must all be able to “...learn, unlearn and relearn.” Let’s use ROMA to unlearn and a few things and to invent some new strategies.
This publication came out of a conversation about “Why are some CAAs having so much trouble with implementation of ROMA?” This workbook attempts to reduce misunderstanding, to get you past a few of the confusing spots and to facilitate implementation of ROMA. This is a temporary document whose usefulness will be eclipsed as CAAs develop capacity and experience and move into ROMA. On the other hand, during the transition, it may save you weeks of wheel-spinning and pointless agony. So – read on!
This publication was assembled by Jim Masters. For the past three years, he has been a consultant to the H.H.S./O.C.S. Monitoring and Assessment Task Force; and on the Survey Subcommittee and the Training and Technical Assistance Subcommittee. However it can not be stated too strongly that the contents of this workbook do not represent the official position of H.H.S. – or anybody else -- other than the author. The author takes full responsibility for any errors – and for the personal opinions included herein.
He used the ideas, input and assistance from the MATF members, State CSBG Agencies, NACAA staff, NASCSP staff and other consultants -- and incorporated their ideas in this publication. Credit is given -- where the author can remember where he got the idea.
Thanks to the following individuals for their feedback on earlier drafts:
Charles McCann, Missouri State CSBG Administrator
Julie Jakopic, NASCSP
Colleen Wagner, NACAA
Send Jim YOUR ideas, or contact him and ask him questions about the ideas in this pamphlet, at: or phone 510.339.3801, or FAX 510.339.3803.
A. The concepts in Results Oriented Management and Accountability (ROMA)
Background.
The first words in ROMA are -- results oriented -- but, if you look at the flow of a program, results come last. They are what happen as a consequence of the operation of the strategy or program.
Inputs Operations Outputs Results/Outcomes
People Program Participants X of this Benefits to person
Money Activity by staff Y of that Benefits to community
Time
So ROMA is an effort to shift our attention away from the inputs and operations of daily activity to the benefits that are produced for the participants and the community as a whole.
Staff are accountable not just for doing their j-o-b, but also for creating the results their job is supposed to produce. Each staff person becomes accountable for producing the results.
Conclusion. These are pretty simple concepts, but are challenging to implement.
Step 1. Start stretching your brain! Expand your thinking past the edges of the old boxes in which we have been working successfully for the past three decades.
The best single one page summary of ROMA that I have seen was written by Scott Anglemyer, from the Kansas CSBG Office. He calls it the "Cliff's Notes" version of ROMA.
1. ROMA is not a set of rules, procedures, and forms. It is a way of thinking about what the community action network does.
2. That way of thinking is essentially that the management of community action agencies should be driven by outcomes.
3. Managing for outcomes involves planning, carrying out those plans, and measuring the results, and using those measurements in the next cycle of planning.
4. The "Results" in ROMA are often discussed as outcomes. When we talk about outcomes, we mean the changes that occur in the lives of families, in communities, or in our agencies as a result of our agencies' activities.
5. Because it is a management system, ROMA is more than just outcomes. But it is outcome driven, and emphasis in workshops and on the ROMA web site tends to be placed on measuring outcomes because that is the part of ROMA that most agencies have the least experience with and knowledge of.
6. States are required to implement ROMA and to report to the H.H.S/O.C.S. on outcomes. Those requirements do not specify what measurements must be used.
7. Most states passed on similar requirements to their grantees ‑‑ they require agencies to measure outcomes, and may establish a general framework, but they usually do not require that specific measures or techniques be used. Therefore, in most states each CAA is free to measure outcomes in the way that makes the most sense for your agency.
8. The things you see on the ROMA web site are examples of measures and measurement techniques that you can use. They are not the only things you can do. OCS has deliberately kept ROMA open‑ended to encourage diversity and innovation across the network.
B. Where did ROMA come from and why should we do it?
Background.
Social values and economic interests translate into public policy anywhere from five to ten years after they have “happened in the streets.” ROMA comes from a huge shift in social values. In the 1980s, public confidence in government was dropping. President Reagan and other political figures began mirroring and adding fuel to the public impression that government was inefficient and ineffective. The business sector was praised as being better at measuring their results -- at describing the bottom line of their activity.
In the late 1980's, the Government Accounting Standards Board promulgated standards on measuring results and costs of government programs. Congress picked these ideas up and incorporated them into the Chief Financial Officers Act of 1990 which required federal agency financial managers to focus more on the results produced by expenditures.
At the same time, board members and managers of many publicly funded programs were also asking how to better measure the results of their efforts, as were state and local officials.
There are many good reasons for doing a better job of describing our results that are given in the ROMA Guide published in April, 1999 by H.H.S./O.C.S. and the Monitoring and Assessment Task Force. Still more reasons are found in the workshop handouts published by The Rensselearville Institute:
* Results matter -- to program participants, staff and funders.
* Partnerships are needed to produce most results. It takes a joint commitment to success.
* New concepts and new words are needed to lead to new thinking.
* People are the single most important piece of the whole equation. People, not plans or money, get things done.
* Planning and doing are inseparable.
* Action, change and learning are keys to success.
* Success means being result-focused.
The social values shifts prompted changes among financial managers and eventually led to new statutory requirements. In 1993, Congress passed the Government Performance and Results Act, (G.P.R.A.) Gyp-ra, as it is called, required creation of strategic plans and reporting on results by all Federal agencies by 1997. And it required that budget requests be submitted using the categories in those plans by the year 2000.
Conclusion. The Government Performance and Results Act of 1993 is one of the key elements prompting the shift toward results and outcomes at the Federal level.
Step 2. Review G.P.R.A. It is a key driver. Use it as a touchstone to assess your efforts.
C. How is G.P.R.A. being implemented outside the CSBG? How are Other Federal Agencies approaching the G.P.R.A.?
Background.
Let’s look at how it is being implemented in DOL, HUD, USDA, and H.H.S.
1. Department of Labor. DOL has a centralization/ decentralization cycle that recurs about every ten years.
The Manpower Development and Training Act of 1962 initially granted substantial authority to states. Over the remainder of that decade, control over programs was slowly centralized back at the Federal level.
The Comprehensive Employment and Training Act of 1973 (CETA) was one of President Nixon’s Special Revenue Sharing Block Grants (along with SSBG and CDBG). Under CETA, local Prime Sponsors (most of which were CAAs) were granted substantial authority over program planning and management. During the 1970's DOL pushed most of the CAAs out of the prime sponsorship role and replaced them with city or county governments. DOL also wrote so many Federal regulations that by the late 1970's the program was not only a bureaucratic nightmare, but in the process of creating it DOL had once again moved into the driver’s seat.
JTPA replaced CETA in 1981, again devolving substantial authority to states and the local PICs. Then, DOL began imposing rigid performance contracts and cost-reimbursement standards in which the SDA could make progress payments for number of people assessed, trained, placed, and still on job after 30, 60 or 90 days, etc. (As an aside, I predict this performance-contracting model will come into human development and public charity services, too.)
In the fourth decentralization cycle in as many decades, Congress has once again delegated substantial authority to the state and local levels through the Workforce Investment Act (WIA). In WIA, responsibility is devolved to states, previous program categories are merged, “one-stop” shopping is encouraged, and the local Workforce Investment Board has broader the representation and a larger role. DOL has developed twelve G.P.R.A. related performance indicators for the WIA, and will eventually negotiate targets on each with the states. The states will then presumably negotiate with each local WIB about what they will accomplish on the 12 performance indicators. Thirteen states are in “fast track” implementation of the WIA and supposedly “started” their new system on July 1, 1999. The other 37 come on board by July 1, 2000.
As this is written, DOL is in the throes of another reorganization. Many top-level career staff are leaving, reducing the capacity of DOL to implement WIA in a timely manner. The upshot of this is that now is the time to work with your State agency and local WIB on what they are going to do and with whom they are going to do it. As one long-time employment and training expert it, “the deck is being shuffled -- don’t wait until all the cards are dealt.”
2. HUD has three new block grants. The new consolidated planning process (ConPlan) developed in the mid 1990's provided a vehicle for HUD to introduce goals and performance measures. HUD is lucky enough to have enough concrete stuff (no pun intended) to count, but as we have seen in the recent Congress, effective program operations, a reinvented and refurbished Federal agency, and substantive results measures are not enough to insulate an agency from the vagaries of the budget process.
3. USDA developed 13 different strategic plans – one for each mission area. The plan for rural development (housing, economic development, facilities development, rural utilities, co-ops) had dozens of NEW results and outcome measures in it. Staff task forces worked over a period of three years in an iterative process to develop and refine their measures.
The 1996 Farm bill required development of strategic plans in each state, and more measures were developed during the planning process in each state. Money continues to flow from both state and Headquarters offices depending on the program. You may want to get involved either in development of the state plan and the performance measures in it and/or in the special task forces attached to the national programs like co-ops and rural utilities.
4. H.H.S.
A. The overall H.H.S plan has five goals. Most of A.C.F. fits under its Goal 2, “Improve economic and social well being of individual, families and communities.”
B. Agency for Children and Families. The A.C.F. has consolidated several plans from the operating agencies and programs including, Head Start, TANF, CSBG, and others. They have created a framework of 4 goals, with 8 program objectives and 2 administrative objectives. They are given next.
Strategic Goal 1: Increase economic independence and productivity for families
1. Increase employment
2. Increase independent living
3. Increase parental responsibility
4. Increase affordable child care
Strategic Goal 2: Improve healthy development, safety and well‑being of children and youth
5. Increase the quality of child care to promote childhood development
6. Improve the health status of children
7. Increase safety, permanency, and well‑being of children and youth
Strategic Goal 3: Increase the health and prosperity of communities and Tribes
8. Build healthy, safe and supportive communities and Tribes
Strategic Goal 4: Build a results‑oriented organization
9. Streamline ACT organizational layers
10. Improve automated data and management systems
Most of the departments in A.C.F. are taking G.P.R.A. very seriously, because the budget submission is now built using the same categories as the plan; i.e. the budget process uses the plan formats to request funds. And, this is exactly what the framers of G.P.R.A. intended.
The H.H.S. web page has the entire A.C.F. plan for the year 2000. It is about 90 pages long. The plan for each coming year is released after the President’s State of the Union Message, so the
2001 plan will be released in February of 2000. It is in draft form and is about 115 pages long, but it is “embargoed” until after the White House has their way with it.
You can download your very own copy from:
On the left of this home page there is a column that includes “performance planning in A.C.F.” Click on it. Then at the bottom of the next page click on “annual performance plans.” And at the bottom of the next page click “here” for the most recent edition, and download it.
Note: The 6 goals for CSBG and the programs managed by OCS fit into the H.H.S./A.C.F. framework. Most CSBG program activity fits under A.C.F. Goal 3, “to increase the health and prosperity of communities.”
The new IDA program fits under A.C.F. Goal 1, “To increase economic independence.”
ROMA as a process relates to Administrative Goal 4, “to build a results oriented organization.”
C. Office of Community Services. The O.C.S. was one of the first Federal agencies “out of the gate” to implement the G.P.R.A. We come back to the approach used by the O.C.S. to develop the 6 goals and measures and ROM, starting on page 18.
D. The Head Start program has developed 25 performance measures of the effects of the program on knowledge and skills acquired by children and parents, and other typical measures. The extent to which Head Start program nationally is meeting most of the standards relating to outcomes for children and parents will be answered through the FACES study. In this national research project, 42 local programs have been selected in a stratified random sample of all 1,700 programs. Interviews and testing of 3,400 families who are participants in those 42 programs will provide the data. In other words, Head Start is answering most questions about their results and outcomes in changes in the children and parents through a research project that is taking place in only 42 of their 1,700 local sponsors. This whole approach is described in their report, Head Start Program Performance Measures, Second Program Report, which is also IM 98-19, issued November 24, 1998.
Information about other aspects of program operations will be answered though the HSFIS system and the PIR’s, which collect information from all programs, but the crucial questions are being done through a research project and a stratified random sample.
E. LIHEAP. They have developed two performance standards so far, and are asking for feedback from the field for additional measures.
F. TANF. I should say something relevant about TANF here. So I will say that I agree with the article that former H.H.S. Counsel Peter Edelman wrote -- after he resigned from H.H.S. in protest along with Mary Jo Bane and David Ellwood -- in Atlantic Magazine titled “The Worst Thing That Bill Clinton Ever Did.” This is an anti-welfare strategy, not an anti- poverty strategy. Yes, the rolls are declining. In 1966 there were 6 million women on public assistance. And between 1966 and 1970 CAAs help another 3 million more women go onto AFDC. This was one of the best things we ever did. So now we are pushing women back into bad relationships, back to their parents, into makeshift living arrangements, and pushing them out of parenthood.
G. Community Economic Development. Most of the Community Development Corporations, including those funded from the O.C.S. discretionary funds, are involved in the Success Measures Project operated by the Development Leadership Network, Baltimore, MD. The created working groups in ten locations around the country, assisted with only $2,000 in funding per site, to try to create outcome measures for the complex community development and community building projects. They struggled with all the same issues address at USDA, HUD and H.H.S. efforts, but did not use much of the experience from the other efforts. It was more of a “whole cloth” re-invent the wheel type of effort, which of course can sometimes produce fresh new tools. The topics areas in which they worked are: Housing, Business and Job Development, community participation, levels of hardship, and the relationship between funders and implementors. This is still very much a work-in-progress and only working drafts are in print. Contact Susan Naimark, PH 617.971.9443. FAX 617.524.2250 or
In the meantime, a very useful tool is already available from the Virginia CAAs. They developed and piloted this in five agencies. It is the “Community Matrix Focusing on Neighborhood Organization and Civic Capital.” It is a type of scale-and-ladder that shows how you can help a community move from dis-organization to higher levels of functioning. This is VERY useful – it is a major contribution to our field. Thanks to the Virginia CAAs for developing this, and thanks to Ted Edlich and others at TAP for making it available to others.
The North Central Regional Central for Rural Development, Iowa State University, also has a useful publication Working Toward Community Goals: Helping Communities Succeed that combines both community development strategy and reporting of results. It came out of a collaboration between the Aspen Institute, Ford Foundation and USDA. Their web page is:
Also, check out the material produced by the Northern California Council for the Community. Their publications are Goals and Success Indicators of Healthy and Self-Sufficient Communities, and Bay Area Partnership: Building Health and Self-Sufficient Community for Economic Prosperity, Status Report. This is based on use of social indicators and is similar to Oregon Benchmarks, but brought down to a community level.
A VERY useful publication to use to look at all the Federal block programs in terms of their purpose, structure, relationship to states is the GAO publication, Grant Programs: Design Features, Shape, Flexibility, Accountability and Performance Information.
Conclusions. Our whole universe is in motion! All Federal agencies are changing their program and reporting structures. You are being asked to “coordinate” and “plan with partners” but Federally funded program structures are changing rapidly.
Step 3. Get used to an increase in ambiguity and anxiety. It is going to be with us for several years.
D. Where can I go for help locally? What is happening in performance and outcome measurement in city and county government, at the United Way?
Background.
The United Way of America has an excellent workbook (five bucks!) on implementing performance measurement systems. It includes an 8-step process with detailed actions to take at each level. It is Measuring Program Outcomes: A Practical Approach. They also publish excellent materials on strategic planning.
The International City Management Association (ICMA) has several publications on performance measurement. One, co-produced with the Urban Institute, is a standard reference work for cities seeking help in measuring the effectiveness of police, fire, solid waste, parks, libraries, transportation, and other municipal services. It is How Effective Are Your Community Services? Procedures for Measuring Their Quality. Another workbook from the ICMA gives a historical perspective, commenting on:
...the resilience of performance measurement as management strategy, noting that a common strand of management thought links the planning-programming-budgeting-system (PPBS) of the 1960's, zero-based budgeting (ZBB) of the 1970's, and management by objectives (MBO) of the 1980's to the current enthusiasm.... As noted by Ehrenhalt, “It is a good idea. It was a good idea in 1943. But it is basically the same idea it was in 1943. It just keeps getting renamed, but...this time, momentum seems to be building....” (Accountability for Performance, p.2.)
Another useful series of publications are from The Evaluation Forum. They have a Field Guide to Outcome-Based Program Evaluation that has a complete overview of designing and conducting an outcome evaluation, and a publication on Managing the Transition to Outcome Based Planning Evaluation that goes into detail on managing the organization dynamics (fear, anxiety, resistance, ambivalence, etc.) involved in the transition.
The Alliance for Redesigning Government at the National Academy of Public Administration is another good source. Try them at:
And if you want to check out the Oregon Benchmarks, go to the Oregon Progress Board at: . And, the National Performance Review, now called the National Partnership for Reinventing Government is at: .
Conclusions: The emphasis on results is taking place at all levels and in all sectors.
Step 4. Find local people to talk to about what they are doing and how they are doing it. There is comfort in numbers -- and maybe some help.
E. Can I go to a local college for help? What about help on outcome and performance measurement from the social sciences? Will my old social science textbooks help?
Background.
Social workers, psychologists and educators have thousands of tests and instruments for measuring individual performance and family functioning. Many of them are “normed” based on giving the same test to hundreds of thousands or even millions of people. A qualified person can administer one of these tests to a person or use it to assess a family and tell you exactly where they stand in relationship to the rest of the population that has been tested on the subjects covered by the instrument.
You can definitely get help from a social scientist on:
* how to make your measures valid and reliable,
* how to construct surveys and do sampling, and
* facilitating your process.
However, if you ask for help on content, you are going to get that person’s subject matter imported into your program. It’s the content they have handy, and it is content in which they have faith. It is the place they will go to when they can’t figure out what else to do. If you hire a social worker to help develop the content of your outcome measures, they will tend to develop measures like those used in social work. If you hire a public health expert to help develop your measures, they will tend to use material from public health. Yet, some people hire a consultant from another discipline on the premise that there is some “pure” scientific methodology they can translated into your field and that they do not bring their content with them. I don’t believe it. I think you will wind up adopting their content.
In the early 1990's, a substantial amount of effort went into trying to identify “risk factors” among youth and to associate these with likely use of drugs. Programs then intervened to reduce these risk factors in the hopes that this would reduce drug use. This is classic example of weak theory that reveals only part of the picture -- followed up with weak (low salience) interventions. One gets tantalizing glimpses of bits and pieces of the reality of youth and drug use, but the intervention programs can not muster the amount of re-parenting and social control needed to change how most youth behave or to persuade them to change how they behave.
If you want to see the incredibly complex discussions in the arenas of mental health and public health, look at Assessment of Performance Measures for Public Health, Substance Abuse and Mental Health, by the National Research Council. This is THE book on performance measures for these subject areas.
Micro-economics, political economy, and rural sociology provide some tools to help agencies and programs measure their results with families and communities. Some very interesting new work is being done by rural sociologists that show the relationship between the social system and economic activity in rural areas. Professors Jan and Cornelia Flora at the North Central Regional Center for Rural Development at Iowa State are leaders in this research. Check out their publications.
Other academic fields, including sociology, anthropology and political science, are very difficult to apply to development of outcome measures. Most of the theory in anthropology, sociology, economics, social psychology, and political science is too general to provide tools for immediate use, and they appear to be years away from developing the practical tools you can use to measure change at the local level.
In the physical sciences they distinguish between “big science” and “small science.” The big science is the grand theories, the small science is the use of those theories to accomplish certain tasks. In the social sciences, we have a lot of big science, but not much small science.
Conclusions. Unfortunately, most of the social sciences provide little assistance in developing the content of outcome measures at the community, family or individual level. If it was easy, you could pick up a book and plug in the measures.
Step 5. Avoid getting lost in the gap between big science and small science. Don’t try to pretend that the small science is there when it really isn’t.
Some
other ideas to avoid this problem are given in Section L, page 40, Approaches
to Developing the Content of Outcome Measures.
F. How is ROMA the same as or different from previous program structures and reporting systems used by CAAs?
Background.
There have been six eras of program structures and reporting systems that were permitted or required by the Federal Government. The elements of those systems relevant to implementing ROMA are described briefly below.
1. 1964-1967. Everybody Does Their Own Thing. The Community Action Program was created with the passage of the Economic Opportunity Act in August, 1964. The largest portion of funding was called “local initiative,” where funds were made available to locally designed programs and projects “that were developed with the maximum feasible participation of the poor.” Initially, local groups sent a letter, a brief narrative and a few budget forms to OEO describing their effort – and they were funded based on that local project description. The reporting of progress and results was done against the material that had been submitted in the application. Reporting was done quarterly, and was largely anecdotal. There were no mandated forms or procedures. The local program structure was whatever the CAA said it was. There were frequent and frantic calls from HQ and the Regional Office to CAAs to determine the number of people being served in various types of programs. Every Congressional inquiry generated a unique data collection process. After only three years of operating like this, it was generally recognized that it was difficult:
* to describe the results of what any one agency did in language that was easily understood by others,
* to summarize the results across agencies,
* to measure quality or progress because there were no generally recognized standards.
2. 1967-1974. Grant Application Process. In 1967, OEO instituted the Grant Application Process, or “GAP.” It was developed by OEO staff, CAA staff, and McKinsey and Company. GAP asked a set of standardized questions about the activity and budget of any project to be funded, although the selection of the purposes and activities of each project were left up to the local agency.
A set of about 150 “program accounts” were developed to create standardized descriptions of what was happening at the local level. For example, Program Account 29 read something like “Adult Basic Education, to assist adults in acquiring reading and writing skills up to the 8th grade level.” Program account 30 read something like “G.E.D. Prep, to help adults get up to the 12th grade level in reading, writing and 1rithmetic – so they could take the GED exam.” In the spirit of local initiative, local programs could still vary their operations for the pre-printed definitions, and at the end of every major category (e.g. education) there was usually a blank called “other” so if you could not shoehorn your description in to one of the existing program accounts, you could make up your own. However, the program account structure made it possible to add up the results of different agencies that were operating “similar” programs. Also, the program accounts often grew into separately funded programs of their own, e.g. foster grandparents, family planning, senior meals, senior employment, etc.
Two new forms, the CAP 84 and CAP 85, asked for detailed information about program activities and program participants. OEO staff were besieged with inquiries from the press and from Congress asking “How many left handed people under five feet tall got services from ABC CAA south of main street in town X last month/quarter/year?” This example exaggerates the type of inquiry a bit, but not much. OEO’s thinking went something like this: “If we require that the CAA report the basic demographic characteristics of every person every CAA comes into contact with, we will be able to answer these questions.” Unfortunately, this was before the age of computers, and boxes and boxes of these forms literally piled up in the halls of the Federal offices, with no way to deal with them. Also, there was little use for the data at the local level. OEO came to it senses and realized that (a) the time it took to collect and compile this was not worth it, and (b) the answers to these types of questions were not found to be useful to CAA staff, and (c) if it was really needed it could be quickly determined from a SAMPLE of the records in an local agency, so OEO stopped requiring the demographic data be collected but continued using the program accounts.
3. 1975-1979. Standards of Effectiveness.