This penguin did a thorough evaluation and concluded that he is short. This was not useful news.

Problem

Your evaluation was expensive, but the results are not very useful.

Solution

Accurately define your stakeholders and their needs, and use that information to develop a well thought-out evaluation plan that focuses on utility and relevance.

If you are evaluating school lunches, is your focus on nutrition, on cost, or on taste?

Discussion

What do Your Stakeholders Need?

Understanding the needs of your stakeholders and determining the purpose of the evaluation is the first step in all evaluation plans. Without knowing whom the evaluation will affect and the needs of the evaluation’s supporters, there is a good chance that your results will be irrelevant. For example, if a superintendent calls for an evaluation of a school’s food services because he is interested in monitoring and improving the nutritional health of the students, providing the superintendent with a cost analysis would not be useful. However, if you talked to a nutritionist and had the food analyzed based on the students’ nutritional needs, you might discover that the food is woefully inadequate. You could then provide the superintendent with a proposal outlining specific meals with comparable costs, thereby increasing the likelihood that the results of your evaluation will be used.

Keep in mind a range of needs for your stakeholders.

Keep in mind an evaluation may have several different stakeholders with competing needs. In the school food services example, the superintendent’s concerns relate to school boards, parents, students, and budgets. In contrast, the food services staff’s concerns probably lean towards the efficiency of food preparation for a large number of students. By talking to each stakeholder, you can determine how you can make the results of the evaluation useful to each.

Are Your Evaluation Goals Actionable?

After you have determined who your stakeholders are, clearly identify your evaluation goals and make sure they are actionable and explain why. For example, an objective for a smoking cessation program evaluation might be the following: “This evaluation will measure the change in participants’ smoking habits over the course of a 90 day program.”

What Data Should You Collect?

Once you have set your goals, you will decide which data you need to collect in order to meet those goals. In the case of the smoking cessation program, you may want to count how many cigarettes participants smoked on the day before the first day of the program and then again after 90 days. In order to do this, you might provide participants with a smoking log that documents, in part, the number of cigarettes they smoked. This can be completed and submitted online at the beginning of the first class.

The results of an evaluation should lead to actionable recommendations. Again, maintain a focus on the utility of the results. While some evaluation (called summative evaluation) is more descriptive in nature, such evaluation can still offer options for acting on the results.

Both summative and formative evaluations are valuable. Which to use depends on which way you want to go.

Summative or Formative Evaluations?

Summative evaluation is used primarily to draw conclusions about the efficacy of a program. The focus of summative evaluation is to describe in order to make a judgment. In schools, summative evaluations are frequently given in order to assign grades to students.

Formative evaluation is a developmental tool used to guide the development of a program. It is utilized at specific points to determine efficacy and make changes based on the results. For example, participants may be asked to offer their feedback on a specific portion of the program. The results of this feedback may be used to inform revisions.

Often, a thorough evaluation requires a mix of formative and summative evaluations. Using formative evaluation regularly can ensure that a summative evaluation indicates a successful program. The main point is that you’ll need to think about the purpose of your evaluation. Do you want to summarize or change your program? Sometimes a program is only offered once. If that’s the case, you may only need a summative evaluation. If it is an annual event, however, you may want to perform a formative evaluation with the intent of increasing attendance the following year.

Unexpected Results: Bogus or Bonus?

Sometimes the results of an evaluation may not appear to be useful because they are not interpreted correctly or because they are unexpected. This may happen when:

* You have not clearly identified your goals.
* You have used inappropriate data collection methods.
* You have used incorrect statistical methods to analyze the significance of the data.

Evaluations are crucial, even when paying for one means scaling down your program.

Unexpected results may still provide valuable information and at times can lead to a complete realignment of a program and its objectives. If, for example, you discover that your program not only helps people stop smoking but also helps them lose weight, the evaluation can significantly impact how you market your program in the future.

You Can’t Afford Not to Evaluate

It’s crucial to include an evaluation in your project, even if it means scaling down the program to access budget resources. In many cases, an evaluation is an absolute requirement – especially for the federal government. Costs can be minimized by keeping the evaluation focused on the core issues of the program. In an educational setting, for example, you can keep the evaluation focused on measuring the students’ retention of the course content. Thoroughly exploring who your stakeholders are and what they need to know will also help leverage the time and money spend on evaluations.