Newsletter

How Nikkei’s testing approach delivered 8x subscription conversions

Nikkei’s clear sense of priorities allowed it to balance the results of its testing programme to increase subscription revenue.

7th June 2024

Nikkei has developed a testing approach that allows it to navigate competing results by focusing on subscription revenue growth. Photo by Tokumeigakarinoaoshima from WikiMedia Commons

As readers of the Media Bulletin know, we’re big fans of data and testing. But what happens when tests deliver mixed results? Leaders have to be the “guardian of trade-offs”, says Michael Porter, whose strategy frameworks are required reading for business school students and business leaders alike. And this week, a case study from Japan’s Nikkei reminded us that successfully navigating these trade-offs can be transformative.

At the AWS London Publishing Symposium, Norihiko Sawa, a Principal Research Engineer at the media group, outlined their approach to testing:

Challenge our assumptions
Design and test our hypotheses
Implement changes based on A/B testing results ASAP

Nikkei published its first newspaper in 1876 and is the world’s largest business newspaper by circulation. However, few Japanese publishers, apart from Nikkei, have expanded beyond their home market. In 2015, Nikkei made its first foray into the global media market when it bought the Financial Times. Norihiko credited the acquisition with adding to their knowledge of subscription operations, and he showed how their approach delivered more subscribers.

When your current paywall model isn’t working

Out of the 120m people in Japan, they consider 12.5% or about 15m as their addressable market of white-collar workers over 20 who live in cities. And 16% of that, or 2.4m, are Nikkei subscribers, with 1.4m print subscribers and 1m digital subscribers.

They have what is fast becoming a standard “marketing funnel” with 25m monthly unique users who only have access to free articles. They have registered 5m users, who can read up to 10 premium articles per month and don’t have to provide payment information. They have 900,000 paying subscribers and free triallists. Paying and trial users have access to the “full subscriber experience”, including access to premium articles and other products such as a popular digital replica edition.

The newspaper, considered the largest business daily in the world, launched its paywall in 2010. Their old paywall was quite porous with half of the content being free to all users, including anonymous users, who could register to access 10 premium articles each month. Norihiko said that it took too long to “build user engagement” and in 2015, when they bought the FT, they only had 400,000 paying digital subscribers, much lower than their print subscriber base.

Finding the ‘sweet spot’ for free articles

In 2019, they launched a new strategy that adopted what is fast becoming a standard model in many markets with the addition of a registration step. The old model delivered low churn but low conversion, while the new model accepted a higher churn rate for higher conversion, something it has in common with the cyclone model, which relies on cut-rate trial offers.

Nikkei relied on free, rather than paid, trials and to make the model work, they tested the number of free articles they provided for registered users. They considered limits of one, two, three and 10 free articles, but they knew from their data under the old model that only 3.19% of users ever reached the 10-article limit each month.

Their working hypothesis was that three free articles would be the “sweet spot” for conversion, and their assumptions were they would see higher conversion rates and the same amount of page views for registered users.

Their approach demonstrates the value of having both a hypothesis to test plus KPIs to test it with, in this case, page views and conversion rate. In their testing regime, having the KPIs in mind added nuance to the interpretation of the test.

After running the test with all free registered users between November 2022 and February 2023, they found no sweet spot for conversion. A limit of only one free article had the highest conversion rate, but it also exerted some downward pressure on page views. A limit of one article had a 1.72% conversion rate, while 3 articles had a 1.44% rate. But their page view index dropped to 75% for a one-article limit but rose to 85% for 3 articles.

They decided to set the limit at one article because as a business conversions were a higher priority than page views, and they found that most of the registered users were inactive anyway, which led to ad revenue earned from registered users being “much lower than from paid subscribers”. The decision led to an 8.4x increase in conversions with a less than 3% decline in monthly active users and a slightly more than 10% drop in page views amongst registered users.

The nuances in paywall CTA tests

Many case studies only discuss the increase in conversions when testing paywall CTAs. However, as with their approach to free article limits, Norihiko spoke about the trade-offs they navigated in settling on the best option.

In their first CTA test, they eliminated Register as a free member from the options they showed so users only saw the option to become a paid member and to log in. As expected, the conversion rate increased by 23%. Yay! However, the churn rate increased from 33% to 49% from the control group of their current CTA and the new one without the option to register. This led to a 6% drop in payment after the trial period had ended, and they decided to keep the three options.

The second hypothesis they wanted to test was to tout the benefits of a subscription in the CTA. They had a control group without the change and three variants, each with a different benefit of subscription including:

access to all premium content
the ability to customise the content they followed
and a premium feature highlights articles based on readers’ interests in a digital replica app.

The digital replica app is very popular amongst their readers, and of the three variants, the ability to highlight articles in it based on their interests was the only one that provided an uplift, a 22% increase in their conversion rate.

Using the same criteria they did in their first test, they found that while churn was slightly higher during the free trial period with the new value-based CTA, 37% to 30%, subscription starts increased by 10%.

How long did all of this take? “We took a lot of time to discuss internally,” Norihiko said, the various tests took one to three months of discussion, one to two months to develop and two months of testing. The upside is that they had sufficient data to be confident with their conclusions, and once they made a decision, they “implement(ed) changes based on A/B testing results ASAP”.

It’s been our experience that when an organisation starts working with data and testing, it takes time for them to navigate cultural barriers, get stakeholder buy-in and develop processes. However, as the tests show results, organisations become more confident with testing, which accelerates business transformation.

Our takeaways from Nikkei’s experience are:

Be willing to challenge your assumptions.
Make sure to have your KPIs set as you start.
Test one thing at a time to isolate cause and effect.
Be patient in collecting data to provide confidence for your actions and quick to act once a decision is made.
As this case study demonstrates, keep your North Star goal in mind to keep a clear sense of your priorities so you can navigate trade-offs.

Industry News

Here are some of the most important headlines about the business of news and publishing as well as strategies and tactics in product management, analytics and audience engagement.