Newsletter

Solving audience ‘pain points’: How publishers can increase their ability to experiment

Rapid changes in the publishing industry mean that the ability to use experiments to deliver innovation has never been more important.

6th October 2023

We discuss a number of ways to foster a culture of experimentation in your publishing business. Image created by Midjourney

In the Pugpig weekly media bulletin, Pugpig’s consulting services director Kevin Anderson and digital growth consultant James Kember distill some of the best strategies and tactics that are driving growth in audiences, revenue and innovation at media businesses around the world.

Helping publishers develop a testing strategy that builds audience engagement and conversion optimisation is what we do at Pugpig Consulting. If you want to know more about how we are working with publishers like you to achieve that, get in touch at info@pugpig.com.

Building the culture and capabilities needed to increase experimentation

During a keynote at the FIPP World Media Congress 2023, Dan Pacheco, Horvitz Chair in Journalism Innovation at the S.I. Newhouse School, Syracuse University, emphasised the need for media companies to innovate. He pointed to the increasingly interactive nature of the internet driving a requirement to continually experiment. For Pacheco, the approach that publishers should take is to identify pain points from within their target market and be the “pain killer”. “If you can do that, you have a solution that’s going to have some longevity to it”, notes Pacheco.

And as we have highlighted, the media environment is changing dramatically. Digiday and Axios flagged up data about how much referral traffic from Facebook and EX-Twitter has declined for both traditional and digital news publishers. Traffic from Facebook has declined by 66% to the New York Times, CNN and Yahoo News. But that is tiny compared to decreases at BuzzFeed (72%), The Daily Mail (77%), The Guardian (79%), Business Insider (80%) and The Sun (84%). Experimentation with tactics to develop new audience acquisition methods as well as efforts at building direct relationships with audiences will be critical to the success and survival of media brands.

But why aren’t publishers already better at experimenting? A report from FT Strategies European GNI Subscriptions Lab, identified two primary blockers stopping publishers from doing more experiments: A poor experimentation culture and deficiencies in tools and capabilities to allow teams to carry out tests.

A poor culture of experimentation

FT Strategies and GNI observed that whilst some publishers would test a lot, that didn’t “mean a culture of experimentation [was] present. They put this down to testing programmes suffering from “a lack of prioritisation, proper iteration, and buy-in from all parts of the business”. This meant that experiments lack a “clear and evidence-based understanding of the problem that needs to be solved”. The result? A lack of systematic ways to share results across the organisation. From our experience, this means that publishers can fail to scale up experiments that show promising results or waste resources by running similar tests across siloed teams or departments.

Deficiencies in tools

This combined with an observed lack of native testing capabilities within critical systems such as the CMS and paywall, and for the news organisations that FT Strategies and GNI worked with, they saw poor connectivity between different parts of publishers’ tech stack. This meant that “most publishers are not able to run tests without significant effort”. For those businesses it greatly reduced their interest in experimenting and meant that many testing opportunities were missed.

How to build the capabilities to test

However, despite the cultural and capability issues that many publishers face, at Pugpig Consulting we’ve worked with several customers to build experimentation roadmaps. So where to start? We would advocate for not trying to boil the ocean. Keep it simple and focus on making small improvements, where results can be achieved reasonably quickly and with statistical significance. A good place to start is with A/B testing and our friends at Piano have published a handy guide that goes into the approach in some detail and which we summarised in a previous edition of the Media Bulletin.

Piano’s testing capability runs counter to FT Strategies and GNI’s observations about the lack of native testing capabilities across critical platforms. Generally what we’ve seen is that the variance between capability can be vast, but a basic level of testing functionality is common within many tools including subscription services, email platforms and push notification providers. Take care to evaluate the ability to test and segment when looking to bring in new tools and consider the wider infrastructure they will sit in. For example, a paywall tool might be able to split an audience into cohorts for testing, but does the data exist to allow that to happen?

To establish technological requirements you should build out a testing roadmap. As Pacheco mentioned, the best way to start is by focusing on the pain points experienced by your customers or internal stakeholders. Answer questions that allow you to address problems that stand in the way of achieving your goals. Decide what tests could help to combat these issues and understand what is possible when. Don’t be afraid to start small and work with a partner if necessary (we can help). If you can prove value delivered or problems solved, you will accumulate the data to support investment in new solutions or products.

Establishing a testing culture

In an article for the Harvard Business Review, Stefan Thomke noted that building a culture of experimentation “takes more than good tools. It takes a complete change of attitude”. Thomke used examples from Booking.com, Netflix, Facebook and Amazon to highlight the need to cultivate curiosity, use data to make decisions, democratise experimentation and be ethically sensitive.

To cultivate curiosity, publishers need to begin rewarding experimentation by shifting focus away from always requiring positive results. Thomke looked at Booking.com where “only about 10% of experiments generate positive results” and whilst it’s understandable that decision makers can be reluctant to allocate resources to tests that might fail, running a high volume of experiments will ultimately take a business to a net positive position. Furthermore, by eliminating options, a publisher can prioritise resources.

To ensure that success can be measured there needs to be an understanding of the “human nature” of the people looking at the test results. A culture of experimentation requires an overall vision with high-level, North Star goals that align the business to key outcomes that then can be tested using quantifiable, empirical, results. Fortunately, the ability of publishers to access data is higher now than ever before. By focusing on business-wide goals and developing data as a common language, publishers can align the editorial, advertising and subscription teams whose goals might at times come into conflict.

Executive support for experimentation does not mean that only those at a certain level within a business should be responsible for testing. The “democratisation” of experimentation is important in building a good testing culture. Effectively, this means encouraging teams throughout the business to suggest and lead testing plans. At Booking.com they built a centralised experimentation function supported by satellite teams. This allowed for anyone in the business to request a test, but it was monitored by a specialist team to mitigate the risk of a drastically negative impact.

As experimentation is rolled out across the business, it also needs to be grounded in ethical practices. It is not simply about respecting GDPR but also about safeguarding that trust that audiences have in your product. It is important to consider the user when data is involved: What would their reaction be if they were aware that a test was carried out? In building of a test culture, encourage debate amongst staff and in socialising of testing frameworks and results.

At Pugpig we’re working with several customers on developing testing frameworks, helping to deliver capabilities, and building a culture that values experimentation. We have seen great results with a 60% increase in engaged users on an app after a push notification testing process and a six times increase in app downloads after testing and refining marketing and onboarding communications. We advocate the use of simple tests in platforms that exist by teams throughout the business, and by holding workshops with not only the staff involved in testing but the wider organisation, it is another step in building your culture of experimentation. In our experience, publishing companies are full of curious people who genuinely care and want to succeed. Experimentation empowers them with knowledge of how to achieve not only the growth of the business but also their own.

Industry News

Here are some of the most important headlines about the business of news and publishing as well as strategies and tactics in product management, analytics and audience engagement.