Measuring and Evaluating Impact
How do we measure our success? Monitoring and evaluation result in sound impact assessment, all of which are critical to the success and integrity of our campaigns and those of our partners. – Anand More, Director Planning / Monitoring and Evaluation, Heroes Project
Heroes Project believes that change can only take place when we monitor, measure, evaluate and report. This is the only way to truly know the effectiveness of communications, advocacy and behaviour change campaigns across multiple channels. Using a rigorous process, we conduct baseline surveys before a campaign to understand the current level of awareness and knowledge of social issues like HIV. Our post-campaign surveys then measure the actual change against our objectives, and the depth and breadth of our impact.
Heroes Project implements broad based communications, advocacy and behaviour change campaigns, from public service announcements such as Mr. Doubt, to HIV awareness episodes on popular TV shows, soaps or mini radio series, to high-powered events with societal leaders.
But questions remain: How do we know that our messages are reaching our target segments as intended? How do we know what to improve, change or do more of?
For our societal leaders strategy on HIV advocacy and communications, we wanted to confirm we had the most effective ambassadors to carry messages into various communities. After all, just because an actor is considered among the most popular in the country, doesn’t mean he or she will be positively received by target audiences for specific issues.
On Heroes Project’s behalf, MaRS Monitoring and Research Systems (MaRS) evaluated societal leaders in Tamil Nadu, Maharashtra, Karnataka, and Andhra Pradesh who were featured in campaigns aimed at female sex workers, men having sex with men, transgendered persons and people living with HIV. The study looked at movie stars, sports celebrities and community leaders.
Meanwhile, with behaviour change campaigns featuring characters like Mr. Doubt, we conduct baseline surveys before a campaign to understand the current level of awareness and knowledge of social issues like HIV. Our post-campaign surveys then measure the actual change against our objectives, and the depth and breadth of our impact. For example, we surveyed movie goers on their recollection of Mr. Doubt and his impact, and discovered that the campaign had prompted many to reconsider their opposition to getting tested for HIV.
Our societal research reveals that in Andhra Pradesh, for example, celebrities do not enjoy much favour among the key target segments as spokespersons in HIV communication. Similarly, male and female movie stars have significant relevance primarily in communication for condom promotion and stigma reduction, respectively. Across all key target segments in Andhra Pradesh, health care providers and community leaders scored above film and TV stars as preferred societal leaders for communication on all other HIV topics.
The results confirmed that societal leaders must be used strategically for specific issues. Moreover, the key populations surveyed demanded that public health providers, including doctors, become societal leaders for the HIV cause, thereby identifying a hitherto untapped genre for effective advocacy.
As a result of this evaluation study, we can now speak with even greater authority to our partners on initiatives such as our societal leaders strategy. We can recommend which societal leaders to include in campaigns, why to include them, and more importantly, what adjustments can be made to current campaigns.
Meanwhile, our research for Mr. Doubt showed a high recall for the character Mr. Doubt and the PSAs’ key messages. Indeed, 98% recalled the key message to get tested for HIV, while 71% recalled Mr. Doubt. More importantly, 72% of respondents perceived the need to get tested and 49% indicated their intent to take the HIV test in the future.
