Evaluation of a civil society-building program for Middle Eastern civil organizations

For the U.S. Department of State (DOS), also in Middle East in 2017, we fielded a team to evaluate a civil society-building program for Middle Eastern civil organizations. From i-APS’s accumulated experience, we knew that a well-executed work plan requires strong internal quality control to ensure high-quality data collection and deliverables. i-APS was contracted alongside Pillar Systems Corporation to evaluate the impacts of a DOS civil society-building program for Middle Eastern civil organizations. The i-APS team consisted of a Program Officer based in Turkey and a Project Director and Executive Director in Washington, D.C. We worked closely with the our partner, Pillar and their team to ensure high-quality data collection and reporting.

Developing and piloting appropriate tools tailored to each assignment and context is paramount for achieving replicable results. For the 2017 US DOS work in Middle East, we rapidly started the evaluation for the DOS by developing survey questionnaires and a KII guide based on the evaluation questions. Following DOS’s approval of draft English and Arabic versions of these tools, during the training of the Middle Eastern Data Collection Team Lead, the word choices and meaning of the Arabic-language materials were scrutinized and debated. The questionnaires and other tools were edited, and subsequently vetted by the enumerators. The KII guide was also modified after piloting. After several iterations, the KII and survey questionnaires were approved by DOS, and survey questionnaires were created in Magpi. After this consultative approach was followed, these survey tools worked faultlessly to collect appropriate data.

Quality control is ensured through the thorough training of every i-APS data collection team. Again, for the i-APS 2017 Department of State’s civil society evaluation in Middle East, we developed a training program and agenda to assist the Data Collection team leader in training the data collection team. The training explained the project to be evaluated and the questions that the evaluation was intended to answer. This also included a step-by-step guide for data collection and the use of Magpi, the tools under development, and samples of poor-quality versus high-quality note-taking from previous KIIs.

Our team was effective in training all the required staff in a calibrated sequence that improved the tools, obtained buy-in and agreement for them from our local team, and thereby established a basis for the survey.

During data collection, we developed a detailed data analysis plan for stakeholder review and approval. Our plan for the DOS civil-society evaluation ensured data reliability and validity by triangulating the qualitative and quantitative data plus other relevant data and literature. Feedback on the capacity-building support provided by DOS via three data sources – from KIIs of senior CSO staff, surveys of non-senior CSO staff, and surveys of beneficiaries — was employed. Key themes were developed from each data source, and findings were derived from at least two out of the three data sources. These findings were delivered to the client in a PowerPoint presentation, allowing for feedback. The Department of State used the final report to make improvements in programming for the following year, and expressed significant satisfaction with our work.

i-APS initiates any study by engaging with local stakeholders to foster positive relations with the community, to facilitate cooperation with data collection in-country, and to increase understanding of the context from which data is being collected.