Measuring Community Engagement: A Case Study from Chicago Public Media

Following an RJI event, Chicago Public Media has developed specific metrics to assess the full range of engagement with the community it serves.

Breeze Richardson

By Breeze Richardson
Director of strategic partnerships at Chicago Public Media (WBEZ-Chicago)

At Chicago Public Media we talk about sourcing the stories we tell from the community we serve. We seek to spark conversations. We strive to be active collaborators with our audience. This is our mission, as a public service, and it is the foundation of the community engagement model authored by RJI Fellow Joy Mayer that explores outreach, conversation, and collaboration.

So when I first spoke with Joy in the Spring of 2011, it was insightful and rewarding to learn so much more about what other media across the country were doing and how our efforts lined up. I attended RJI's “The Engagement Metric” conference in May 2011. Following that experience, Chicago Public Media launched an inaugural slate of community engagement metrics.

In the report that resulted from the Engagement Metric conference, it was eloquently put: “… as news organizations fight for survival, a more connected relationship with communities should be valued, and therefore measured.” At Chicago Public Media, I have identified five areas of measurement to give numbers to our efforts at community engagement. These are explained in detail below, but in short they encompass:

• Station partnerships and collaborations,
• Audience commenting online,
• Audience participation in content creation,
• Audience recommending content online, and
• The production of station events.

Though we have only just begun, by tracking these metrics over time we hope to gain a better understanding of what works and why. Our goal is to demonstrate how engagement impacts civic health and increases the local significance of the station as a community institution. It is also our hope that this case study is helpful in furthering the industry’s collective understanding of community engagement initiatives and their impact, and that over time cooperation to share engagement numbers – as the industry does with listener and sales data – will one day be commonplace.

Metric 1: Total number of Chicago Public Media partnerships, the number of collaborations with partners, and the total collaboration-based content output.

Personally, I am deeply invested in Chicago Public Media’s notion of partnerships. This long-term commitment to collaborate with cultural institutions increases community engagement opportunities throughout our region and provides content for radio, digital, and event platforms. Over the past five years, we have had the benefit of exploring partnerships with a wide range of community institutions and know how our resources have been increased when a partner organization is also contributing staff time, tangible resources, audience, and promotion.

The numbers: In the month of September, Chicago Public Media reported 72 formal institutional Partners, 39 collaborations taking place with these Partners during this month, resulting in 72 pieces of content. This included:

• 37 Chicago Amplified story posts,
• 5 Dynamic Range feature story posts,
• 5 broadcast stories as part of WBEZ’s Changing Gears Local Journalism Initiative,
• 6 broadcast stories collaboratively produced with a Partner organization and distributed as a series titled “Ground Shifters” on Worldview,
• 8 pieces of content broadcast on Vocalo/89.5 FM,
• 2 public events hosted at WBEZ’s Community Bureaus,
• 5 additional co-produced public events titled “Marking the Day: The 10th Anniversary of 9-11”, and
• 4 public events where WBEZ served as the media sponsor.

Three months in, we can already see in the numbers things we anecdotally knew: for example summer months have less activity, and more collaboration opportunities yield more content. But perhaps the biggest reward is seeing the breadth of this work all documented in one place and easy to share with our Board, with colleagues, and with key decision-makers thinking about staffing and resources.

In addition to engagement on an institutional level, I also looked for ways to capture a better measurement of the daily engagement going on directly with the listeners and users to our radio stations and websites. By utilizing existing tools we found ways to count comments to our stories, participation in our production work, and the sharing of online stories by our audience.

Metric 2: “Consume:Converse Ratio” to measure the number of page views to story-level content (only), per single comment published.

A direct recommendation from the Engagement Metric conference, tracking the number of comments as successful community engagement makes complete sense. What better indicator of conversation than audience members presenting their views publicly? By tracking this information as a ratio, the number of comments in a given month is put in relationship to the number of page views to story-level content only: story posts, episode-segments from broadcast programs, blog posts, and event listings.

The numbers: In the month of September, the WBEZ Consume:Converse Ratio was about 400:1.

Is this number good? Without anything to compare it to, we don’t really know. But after tracking it for a while, I trust a conversation will take place to explore whether we think it is. If not, strategies can be adopted to improve it, and the impact of those can be tracked over time. If so, we can work to maintain this ratio as WBEZ.org grows and changes.

I recently surveyed several editors in the WBEZ CityRoom to learn more about their experiences with public comments. One spoke of the importance that anonymity plays in welcoming those who can’t or won’t participate publicly, for example when WBEZ published a story about questionable police tactics caught on tape. Another spoke of extending the questions being asked editorially to the general audience, for example as “Occupy Chicago” looks for a permanent home.

“Sitting around the table at an editorial meeting, we realized it was actually a hard question due to city laws, and we didn’t know the answer... so what better way to explore the question than by asking anyone to answer it?” said WBEZ Content Development Director Steve Edwards. The goal, he went on to say, is to remain value neutral and simply ask the question.

Metric 3: Total number of C.U.E. Opportunities, total number of C.U.E. Content (i.e., Call-in, Upload and E-communication)

Anyone who has spoken with me about measuring community engagement knows the frustration I’ve felt. The majority of impact assessment has been project-specific and in no way comparable across projects or initiatives. More to the point, these efforts analyzed specific engagement strategies, but not the long-term overall community engagement investment of an institution over time. Seeking to change this, it seemed the opportunities for audience participation created by station producers and reporters could be grouped into three categories: Call-in shows, Upload projects (i.e. crowd-sourcing), and solicited E-Communications (asking for production assistance by inviting audience to suggest questions, topics or guests).

The numbers: In the month of September, WBEZ reported 19 C.U.E. opportunities and 432 instances of C.U.E. content. This included: 5 call-in segments/episodes which reported 156 calls, 4 upload projects which reported 191 uploads, and 10 solicitations for guest or question suggestion which resulted in 87 submissions.

Tracking both the number of these opportunities created by station staff in a given month and the number of audience participations in them, we can track the station’s engagement work in aggregate. And important to the future of this work, I believe tracking what has been done also increases confidence in our producers and reporters to do more; showing past participation indicates that audience wants these opportunities and will participate again. It also lowers the risk experienced by staff when proposing the inclusion of such opportunities in future programs and initiatives to editors and colleagues.

One recent WBEZ special project, Out of the Shadows, explored the fractures in mental health care for children in Illinois and included a call-in show in the series’ initial plans.

“I knew seeking adequate care – who are the good doctors, helpful agencies, etc. – was one of the biggest concerns parents’ have” said Senior Producer Aurora Aguilar. She wanted them to be able to simply call-in and ask, and accompanied the live radio broadcast with a live online chat, which staff says increased the success of the call-in show. Allowing audience participation in the questioning appears to deepen everyone’s learning – staff included – particularly when specific examples or points based on the questioner’s experiences are raised.

Metric 4: Total number of shared stories; aggregate of all current sharing tools on WBEZ.org.

As we explore ‘relevance’ as a guiding principle in our work, tracking the number of stories that are shared via any of the sharing tools installed on our website can be seen as an indicator of the relevance these stories have for our audience. We are able to acquire the aggregate number across all current sharing tools through Google Analytics Event Tracking, and by counting the number of stories shared online – whether it be a “Recommend” link on Facebook or emailing the story to someone through the provided link – we gain a digital metric which feels akin to measuring how often our audience shares something heard on public radio at a dinner party, a commonly used sentiment to suggest one is a fan. Measuring online activity in this area seems key and potentially more influential than the historic dinner party comment because it sends the recipient directly to the source content online, potentially expanding audience while showing our influence.

The numbers: In the month of September, about 2,600 pieces of content on WBEZ.org were shared. (During this time, WBEZ.org as a whole received 281,580 views to content pages.) Every act of sharing counts as one, with top sharing platforms being reported as Facebook, Twitter, Email (via a web form), and Google Gmail - all embedded tools on WBEZ.org to enable content sharing. In the future, Chicago Public Media aims to track this type of engagement on other platforms and services.

Metric 5: Number of events produced, total attendance (aggregated), percent total capacity.

Bringing together audience to see each other and dynamically contribute to the ideas and conversations sparked by Chicago Public Media programs and initiatives is a valuable piece of community engagement work. Beginning in 2008, we decided to launch an annual event series – now titled the Off-Air Event Series – which provides one event per month covering a wide range of topics. Since launch, over 20,000 people have attended an Off-Air Series event, which complements other station event production ranging from Member Meet-Ups to movie nights and book discussions.

The numbers: In the month of September, 8 events were produced with total attendance of 788 (70% total capacity). To-date, this made 22 events produced since July 1 with total attendance of 3,559 (68% capacity).

In conclusion, as Chicago Public Media moves forward with this work and its measurement, I do feel there has already been a transformation in the value community engagement is perceived to have. As someone who champions this work, that alone feels like a huge victory. One editor said what we can gain as a staff from all this is an assessment of best practices not just best intentions, that audience participation helps us broaden our agenda, and that our best experiences come when we know our objective with the audience by asking when engagement within the production process is going to be most helpful.

As these questions are asked and answered, I have no doubt it will help this institution meet its mission to better connect residents of our region to one another in a relevant and evocative way. And I look forward to conversations continuing throughout our industry around strategies and successful initiatives to do this work, including how we can best measure it.

• Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
• Lines and paragraphs break automatically.
• Filtered words will be replaced with the filtered version of the word.
• Web page addresses and e-mail addresses turn into links automatically.