Program evaluation

Program evaluation is a formalized approach to studying the goals, processes, and impacts of projects, policies and programs. Program evaluation is used in the public and private sector and is taught in numerous universities. Evaluation became particularly relevant in the U.S. in the 1960s during the period of the Great Society social programs associated with the Kennedy and Johnson administrations. Extraordinary sums were invested in social programs, but the impacts of these investments were largely unknown.

Program evaluations can involve quantitative methods of social research or qualitative methods or both. People who do program evaluation come from many different backgrounds: sociology, psychology, economics, social work. Some graduate schools also have specific training programs for program evaluation.

Dimensions of Program Evaluation

Program evaluators may assess programs on several dimensions to determine whether the program works. Rossi et al. (2004) divide these dimensions into 5 main categories: needs assessment, program theory, process analysis, impact analysis, and cost-benefit & cost-effectiveness analysis. [Rossi, P.H., Lipsey, M.W. & Freeman, H.E. (2004) "Evaluation: A Systematic Approach". ]

A needs assessment examines the nature of the problem that the program is meant to address. This includes evaluating who is affected by the problem, how wide-spread the problem is, and what effects stem from the problem. For example, for a housing program aimed at mitigating homelessness, a program evaluator may want to find out how many people are homeless in a given geographic area and what their demographics are.

The program theory is the formal description of the program's concept and design. This is also called a logic model or impact pathways. The program theory breaks down the components of the program and shows anticipated short- and long-term effects. An analysis of the program theory examines how the program is organized and how that organization will lead to desired outcomes. It will also reveal unintended or unforeseen consequences of a program, both positive and negative. The program theory drives the hypotheses to test for impact evaluation. Developing a logic model can also build common understanding amongst program staff and stakeholders (see Participatory Impact Pathways Analysis).

Process analysis looks beyond the theory of what the program is supposed to do and instead evaluates how the program is being implemented. The evaluation determines whether target populations are being reached, people are receiving the intended services, staff are adequately qualified, etc.

The impact evaluation determines the causal effects of the program. More information about impact evaluation is found under the heading 'Determining Causation'.

Finally, cost-benefit or cost-effectiveness analysis assesses the efficiency of a program. Evaluators outline the benefits and cost of the program for comparison. An efficient program has a lower cost-benefit ratio.

Determining Causation

Perhaps the most difficult part of evaluation is determining whether the program itself is causing observed impacts. Events or processes outside of the program may be the real cause of the observed outcome (or the real prevention of the anticipated outcome).

Causation is difficult to determine. One main reason for this is self selection bias. People select themselves to participate in a program. For example, in a job training program, some people decide to participate and others do not. Those who do participate may differ from those who do not in important ways. They may be more determined to find a job or have better support resources. These characteristics may actually be causing the observed outcome of increased employment, not the job training program.

If programs could use random assignment, then they could determine causation. A program could randomly assign people to participate or to not participate in the program, eliminating self-selection bias. Thus, the group of people who participate would be the same as the group who did not participate.

However, since most programs cannot use random assignment, causation cannot be determined. Impact analysis can still provide useful information. For example, the outcomes of the program can be described. Thus the evaluation can describe that people who participated in the program were more likely to experience a given outcome than people who did not participate.

If the program is fairly large, and there is enough data, statistical analysis can be used to make a reasonable case for the program by showing, for example, that other causes are unlikely.

Types of Program Evaluation

Program evaluation is often divided into types of evaluation.

Formative Evaluation occurs early in the program. The results are used to decide how the program is delivered, or what form the program will take. For example, an exercise program for elderly adults would seek to learn what activities are motivating and interesting to this group. These activities would then be included in the program.

Process Evaluation is concerned with how the program is delivered. It deals with things such as when the program activities occur, where they occur, and who delivers them. In other words, it asks the question: Is the program being delivered as intended? An effective program may not yield desired results if it is not delivered properly.

Outcome Evaluation addresses the question of what are the results. It is common to speak of short-term outcomes and long-term outcomes. For example, in an exercise program, a short-term outcome could be a change knowledge about the health effects of exercise, or it could be a change in exercise behavior. A long-term outcome could be less likelihood of dying from heart disease.

CDC framework

In 1999, the Centers for Disease Control and Prevention (CDC) published a six-step framework for conducting evaluation of public health programs. The publication of the framework is a result of the increased emphasis on program evaluation of government programs in the US. The six steps are:
# Engage stakeholders
# Describe the program.
# Focus the evaluation.
# Gather credible evidence.
# Justify conclusions.
# Ensure use and share lessons learned.

See also

* Impact assessment
* Impact evaluation
* Important publications in program evaluation
* Participatory Impact Pathways Analysis
* Policy analysis
* Policy studies

External links

* [http://www.eval.org/ American Evaluation Association] Includes a link to [http://www.eval.org/EvaluationLinks/default.htm Evaluation resources] such as organizations, links sites, email discussion lists, consultants and more
* [http://www.evaluationcanada.ca/ Canadian Evaluation Society] Includes a link to [http://evaluationcanada.ca/site.cgi?section=1&ssection=1&_lang=an Evaluation information] such as services, professional development, resources, organizations, regional chapters
* [http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm CDC six-step framework.]
* [http://www.evaluationwiki.org The EvaluationWiki] - The mission of EvaluationWiki is to make freely available a compendium of up-to-date information and resources to everyone involved in the science and practice of evaluation. The EvaluationWiki is presented by the non-profit [http://www.evaluationwiki.org/wiki/index.php/Evaluation_Resource_Institute Evaluation Resource Institute] .
* [http://www.internationalevaluation.com/ International Organization for Cooperation in Evaluation]
* [http://www.innonet.org Innovation Network] Innovation Network is a nonprofit organization working to share planning and evaluation tools and know-how. The organization provides online tools, consulting, and training, for nonprofits and funders.
* [http://www.education.purdue.edu/AssessmentCouncil/Links/Index.htm Links to Assessment and Evaluation Resources] List of links to resources on several topics, including: Centers; Community building; Education and training in evaluation; Foundations; Indiana government & organizations; Links collected by...; Logic models; Performance assessment & electronic portfolios; Political & private groups or companies; Professional assns, orgs & pubs; Purdue University; United States Government; Web searches for publications by author & topic; and Vivisimo topical meta searches.
* [http://www.maine.gov/legis/opega/ Maine Legislature's Office of Program Evaluation & Government Accountability] An excellent example of a governmental Program Evaluation office with links to several detailed reports which include methodology, evaluations results, recommendations and action plans.
* [http://www.ncsl.org/nlpes/ National Legislative Program Evaluation Society] Includes links to state offices of program evaluation and/or performance auditing in the USA
* [http://www.upjohninstitute.org/ W.E. Upjohn Institute for Employment Research]

References