Помоги проекту - поделись книгой:
• donor organizations increasingly require NGOs to submit analytical reports;
• at the annual conferences of the Association of Specialists in Program and Policy Evaluation (ASPPE)[29], there has been a noticeable increase in the number of presentations on experiences in developing internal evaluation and its importance for high-quality organization management.
Six years ago, as I was observing these processes and comparing them to what was happening abroad, I suggested that by 2026 Russia would see “an increase in the evaluation capacity of various organizations, the development of internal evaluation and self-evaluation” (Kuzmin, 2016). Looks like this prediction is coming true. Another proof of this is the creation of the PROOCENKU Alliance[30] in 2021 and the opening of an online discussion club of the same name, focused mainly on the development of internal evaluation and self-evaluation:
“The PROOCENKU Club is an ongoing online discussion platform for anyone interested in evaluation. The Club’s slogan goes as follows: “An interesting conversation in a good company.” The Club’s mission is to support a community of practitioners dedicated to integrating evaluation into the work of socially oriented organizations. Nothing is required to participate in the meetings, other than the desire. Education, level of training in evaluation, and practical experience are not important.”[31]
ASPPE has developed and published a number of important works on the professional development of professional evaluators. These include the Principles of Program and Policy Evaluation (ASPPE, 2017).
Professional principles are born out of practice and determine how members of the profession should behave in situations of difficult choices. Principles cannot be invented before real experience has been accumulated: it is a natural evolutionary process. When a profession reaches a certain level of maturity and a professional community (association) has been established, that community comes together to agree on the principles. Therefore, the principles of program and policy evaluation adopted by ASPPE are important not only as guidelines for action, but also as evidence of the emerging evaluation profession in Russia.
These principles are designed to take into account the fact that, generally, three parties can participate in the evaluation:
• The Customer is a representative of the organization that initiates the evaluation, orders the task to be implemented, and will be the main user of the evaluation results.
• An evaluator – a specialist hired by the Customer to perform the evaluation.
• Evaluation participants – the people who provide the information for the evaluation; these are usually employees or managers of the programs being evaluated (ASPPE, 2017).
Figure 1. Self-evaluation: three in one
ASPPE suggests that evaluation must be guided by the following principles:
1. Focus on practical use of the results.
2. Competence of the performers.
3. Appropriate methodology.
4. Transparency.
5. Safety.
6. Flexibility.
The “general case,” for which ASPPE principles were developed, is an external evaluation (see Figure 1).
A program (project) self-evaluation is the systematic collection of information about the activities of the program (project), its characteristics and outcomes, which is performed by the
Do the ASPPE evaluation principles apply in a self-evaluation situation?
We had a series of discussions on this issue as part of the 2022 PROOCENKU Club meetings[32]. They were attended by more than 40 people, mostly coming from non-profit organizations from many regions of Russia.
The outcome of these discussions can be summarized as follows:
1. The evaluation principles proposed by ASPPE are applicable to NGO self-evaluation.
2. Short definitions of these principles also apply to the self-evaluation situation.
3. However, recommendations for the use of these principles in self-evaluation practice require significant adjustment and simplification.
We have developed and discussed guidelines for a self-evaluation situation. This is how the
Recommendations:
• You need to be as specific as possible as to why you are doing self-evaluation. Think carefully and discuss the scope of work for self-evaluation. Formulate the questions you want answered, and discuss who will use those answers and how.
• If it is not clear who and how will use the answers to any of the suggested questions, exclude these questions from the scope.
• Include only those questions where answers are not known or at least not obvious to you.
• See if there is a simpler way to answer these questions, rather than performing a self-evaluation.
• Try to keep the number of questions to a minimum. Leave only those of most importance to you.
• At the end of self-evaluation, hold a meeting to discuss the results and plan actions required to use those results (with timelines and responsibilities).
• Think about whether some of the results of your self-evaluation can be useful to someone else besides you.
Recommendations:
• Self-evaluation is carried out according to certain rules. You don’t have to become an expert evaluator, but you do need to know the basics. To do this, you can read special literature or send some of your employees to a training.
• Keep in mind that it is always possible to consult with experts in areas where your own knowledge and skills are lacking. For example, your local university or professional evaluators association.
• Consider self-evaluation an opportunity to learn through practice. Discuss the experience and learn from it.
Recommendations:
• When planning self-evaluation, consider the limitations of available resources and time. Remember that you do not have all the necessary knowledge and skills for evaluation.
• It is important to have an understanding of the strengths and limitations of different approaches to project and program evaluation.
• When choosing self-evaluation methods and tools, it is better to do simple things right than to do complicated things wrong.
• If you have any doubts about the capabilities or appropriateness of a particular tool, you should consult with an expert or refrain from using that tool.
• Remember that the familiarity and prevalence of techniques do not guarantee that they will be appropriate for self-evaluation of a particular project in a given setting.
• Everyone can make mistakes, and it is important to learn from them and avoid their repeating.
Recommendations:
• Make sure that everyone involved in self-evaluation has sufficient information about the process and agrees to provide data voluntarily (respecting the principle of “informed consent”).
• Full results of the self-evaluation are only intended for you. For everyone else, they can either be partially open (at your discretion) or completely closed.
• You need to consider that you may have a bias towards your own project and be suffering from having lost the fresh perception of the project (“blurred view”): these issues should be openly discussed by the self-evaluation participants.
• Since you are performing the self-evaluation for yourself, not for someone else, you are interested in seeing your project for what it is. In this situation, embellishing simply makes no sense. Therefore, you should not view the self-evaluation situation as a potential conflict of interest or be concerned about it.
Recommendations:
• Discuss whether or not self-evaluation could result in harming someone. If this is a possibility, even in theory, all participants should be made aware about it.
• Discuss how you can minimize the risk of negative consequences of self-evaluation. Do your best to avoid such consequences.
• Self-evaluation should be conducted when NGO employees are in a resourceful state; that is, when they have enough emotional, physical, mental strength and energy to do so. Participating in self-evaluations is an additional burden for the employees, and this should be considered when planning their work, to avoid burnout.
• Self-evaluations are part of your organization’s activities, so they must be conducted in strict compliance with the ethical standards relevant to your NGO’s scope of work.
Recommendations:
• There are many different approaches to evaluating projects and programs. You should choose the one that is best for your project, given the conditions. Even if something has worked well for evaluating your projects in the past, it does not necessarily mean it will work for your current project.
• In the process, be flexible and adjust the plan, methods, and tools of self-evaluation as needed to accommodate changing conditions or newly emerging factors.
• Just in case, check yourself: isn’t the decision to be “flexible” an excuse for not conducting self-evaluation in good faith, and could it lead to negative consequences?
Self-evaluation involves building one’s capacity for evaluation. In this regard, it may be useful for NGOs to keep in mind the following “formula” (Kuzmin, 2009):
According to this “formula,” if you know how to do an evaluation but don’t do it, the capacity is zero. Similarly, if you do an evaluation but don’t know how to do it, the result is still zero.
In other words, capacity development in evaluation involves balancing two interrelated tasks: one must both learn and apply the knowledge and skills gained in practice. The principles of self-evaluation take both factors into account.
Поделиться книгой: