Over the years I’ve been involved in PSIAs that seemed to make a real contribution to a project/policy and those that didn’t (or where it certainly wasn’t obvious).
Mulling over how to get the most out of a PSIA, I’ve compiled 6 tips, of a fairly practical nature. These strike me as being the basics. What you need to ensure if your PSIA has decent shot at informing policy. I’m not really touching on research here - of course, the quality and rigor of your methods are of critical importance. Do you agree with the items in this list? Is something missing? What has your experience in this area?
1) Pull together a good team: A good PSIA study team brings diverse skills and backgrounds (and of course professional competence and good interpersonal skills). Being able to draw on the range of skillsets as per the diagram pushes up the quality of analysis. Brainstorming sessions are more productive. The work is enhanced by the cross-fertilization of ideas. As a bonus, team members learn new skills from each another:
- Management skills
- Quantitative methods
- Qualitative methods
- Sector expertise
- Country knowledge
This doesn’t mean a team of seven or more people is necessary. One person usually brings multiple skills to the party. Typically, PSIA teams comprise three to five members, plus a local research firm to conduct primary data collection. Low-budget PSIAs (<$100,000) will struggle to afford a team like this. However, the lack of diverse skills can be compensated for to a degree through close consultation with specialists, e.g. academics, or sector specialists at the donor organization. Formal quality review and peer review processes can also be used to leverage the quality of the outputs.
2) Use both quantitative with qualitative methods to get the full picture: Different methods of data collection and analysis complement each other. They can compensate for each other’s respective weaknesses, thereby deepening the analysis. While PSIAs can be done using only household data, for example, they risk missing important contextual information, the ‘how’ and ‘why’. PSIAs conducted using only qualitative methods (such as focus groups, in-depth interviews) won’t speak to how representative the collected information is.
3) Work closely with the data collection firm. Primary data collection will invariably be conducted by a local research firm. Work closely with them. I cannot stress this enough. Experience shows that it is not enough to draft a TOR and let the firm go off and do its merry work, while merely checking in periodically. The PSIA team should sit down and collaborate closely with the firm to develop and revise the research instruments, pilot the field work, do data quality control, review the data output for internal validity, and do a spot check of questionnaires or focus group output. Unless the firm has excellent and proven analytical capacity (a rare thing in many countries), do the analysis and report writing yourself. Research firms are businesses and, if you’re lucky, they will deliver to TOR specifications. They are simply not going to spend time grappling with the big issues, questioning assumptions, or agonizing over report structure. That is your job.
4) The process matters as much as the analysis: A PSIA’s usefulness increases when the process is given as much attention as the analysis. Agreeing with other stakeholders on the key questions, discussing methodology, conducting interviews, sharing preliminary results – all these activities are opportunities to build rapport with policy makers. At the end of the day, you want to generate buy-in for the final results and recommendations. (Buy-in is not the same as impact, but it’s a big step in the right direction.)The PSIA will get more traction if attention is paid here. A theory of change for promoting buy-in of policy research might look like this: regular communication è relationship è trust è buy-in
5) Report to a working group or advisory committee: Ideally, a PSIA should be accountable to a broader grouping of stakeholders, comprising representatives from relevant government agencies, civil society and the private sector (if applicable). Policy makers in the direct counterpart agency (e.g. the sector ministry or agency, or Ministry of Economy) should organize and chair such a group, which then meets three-four times over the course of the study process and dissemination. The existence of a working group adds a degree of formality to the process, which helps raise awareness and support for the findings. If an appropriate working group already exists, use it.
6) Allow enough time, but be timely: Findings should feed into the policy design and policy dialogue while the window of opportunity is open. Start discussions around the study early and keep them going. I’ve been involved in PSIAs which were delayed for months because of the need for internal approval of a Concept Note, for administrative reasons, etc. As a result, there wasn’t much time to let the policy dialogue ‘bloom’, and the reporting was rushed. A PSIA that comes too late to inform project or policy design will have less value. You also don’t need to wait until the analysis is completed and the report is finished before introducing findings into the policy process. Engaging in consultations with policy makers from day one, discussing the key questions, and sharing preliminary results are all valuable ways of increasing a PSIAs’ impact before it is completed.