Tag: pros and cons

Hopefully, after reading these last few posts about how great evaluation is, you’re totally convinced and have decided that you need it. If so, you may be thinking, “OK, what’s next? How does this actually happen?”

First, you need to decide who should do the bulk of the work: you or someone else. In more fancy terms, this is “internal evaluation” and “external evaluation”. There’s pros and cons to both.

Internal Evaluation (aka “do it yourself”)

Remember that you need to have staff or volunteers who have available bandwidth for this option… don’t overload your own team!
Pros:

• Cheaper: Hiring an external evaluator often means someone HARC with lots of graduate education and years of expertise, and that doesn’t come cheap (nothing good does!). Your internal team probably comes cheaper than that, and they don’t have to travel.
• Doesn’t require collaboration: This makes the process faster—if you don’t have to spend time explaining your program to an outsider, collaborating with another team, etc., things can move pretty quickly. There’s also no real possibility for miscommunication if it’s just you doing the work (at least I hope you don’t miscommunicate with yourself).

Cons:

• Lack of expertise: You have great expertise in your particular field, whether that’s teaching kids, managing disease, or reducing crime. However, chances are, you don’t have a lot of expertise in evaluation, and may make some basic mistakes.
• Perceived lack of objectivity: You strive to be objective. But to outsiders, if you do your own evaluation, it may just sound like you’re patting yourself on the back.
• Lack of “outside the box” thinking: You’ve been so immersed in this program for so long, that you may not be attuned to other emerging outcomes that have been happening, and so might miss these.

External evaluation (aka “let’s hire someone else!”)
Not surprisingly, the pros and cons of external evaluation are mirror images of the pros and cons of internal evaluation.

Pros:

• Conducted by experts: Evaluators (like myself) did actually go to school to learn how to do this, and do have years of experience. So while you are the subject matter expert, I’m an evaluation expert. We work together to make evaluating your program the best it can possibly be.
• Perceived objectivity: Having a third-party do your evaluation is like a stamp of approval. People tend to take the results more seriously.
• Outside-the-box perspective: Being one step removed, evaluators can see changes that have happened that might have gone unnoticed (or at least unmeasured) by you and your team.

Cons:

• Expensive: A good evaluator doesn’t come cheap, and you get what you pay for.
• Requires collaboration: Collaboration is awesome when done right, but it does take time and effort on both parties, and there can be miscommunications between two teams just getting to know each other.

So, how to pick?
There’s no right or wrong answer—it depends on your capabilities, resources, and the subject matter. But here’s some things to consider.

If you choose internal evaluation, consider hiring an evaluation consultant to advise you. This will be cheaper than a full-on external evaluation, but will retain many of the positive aspects like expertise and “outside the box” thinking. You’ll still retain internal control over the process, and it’ll be more affordable than having a full external evaluation. Here’s some services you might want to ask an evaluation consultant to help you with:

• Advice on creating a strong evaluation plan for a major proposal you’re submitting
• Initial meetings to help you identify your desired outcomes and how to measure progress towards those goals
• Recommendations on methods or instrument choice
• Proofing of statistics and/or reports

If you choose external evaluation, be sure to hire the right evaluator for the job. Don’t hire just anybody (or the lowest bidder). Take into account things like:

• Expertise: Do they have experience evaluating projects like yours, or projects on similar subjects? It’s not 100% necessary (evaluation is evaluation, after all), but it’s nice to see work samples and to know they are familiar with the language and concepts of your work.
• Experience: Do they have a track record? Again, not saying you should never hire the newbie (if we did that, how would we ever get to have experience?!), but it’s nice to have someone with a wide variety of techniques under their belt.
• Rapport: A good external evaluation involves a lot of collaboration. You will by no means be hands-off in this process (if you are, there’s something wrong). So don’t hire someone you don’t want to work closely with. It sounds like a little thing, but it’s actually important.

If you do want to hire an evaluator or evaluation consultant, you should hire me/HARC. The end.

No, not really… Much as I want to keep you all to myself, I know that it’s possibly not the best fit for each and every one of you. To find an evaluator other than me/HARC, I’d recommend going to www.eval.org; it’s the website of the American Evaluation Association. They have a free “Find an Evaluator” function where you can search by state (remember, local is almost always cheaper!) or by keyword to get someone with the expertise you’re looking for.

That’s it for evaluation for now… I promise that next week I will move on to needs assessments (complete with practical examples), and how they change lives!