Newsletter

Why keeping an eye on your crawl budget is a key element of your SEO strategy

Good keywords are only one aspect of an SEO strategy. We review how to manage your crawl budget to keep your content high in search rankings.

16th March 2023

Google Search Console is one tool that you can use to check your search budget. Photo by Justin Morgan from Unsplash.

In the Pugpig weekly media bulletin, Pugpig’s consulting services director Kevin Anderson and digital growth consultant James Kember distills some of the best strategies and tactics that are driving growth in audiences, revenue and innovation at media businesses around the world.

How to manage your site’s crawl budget to ensure content discovery

As the digital media landscape continues to expand, ensuring that content is discoverable, relevant and up to date is a key driver of traffic to your site or app. Without a top of funnel content strategy it is very difficult to attract and engage users so you are able to generate revenue from advertising, subscription, membership, events, e-commerce and more. This means that in addition to having good keywords as a part of your SEO strategy you also need a robust technical approach as well.

This week in the WTF is SEO newsletter, Barry Adams talks about one aspect of technical SEO that every news publisher, or indeed any publisher with a large site, needs to be aware of: the crawl budget. Even with the incredible resources of Google, the internet is so vast that it and other search engines need to ration how much time they spend indexing any site, hence a crawl budget. That means that publishers need to make sure that content is optimised for it to get crawled comprehensively.

Sign up to get the Media Bulletin in your inbox.

The Google Search Console is the starting point for seeing where the crawl budget is being spent. At the highest level, the console displays total crawl requests, total download size and average response time. The most important of these is the average response time which shows how quickly the website responds to the Google Searchbot. If the crawler is waiting for the website to load, it’s using up time that should be spent crawling. Adams suggests a response time of 600ms or faster, and he suggests using a CDN – a content delivery network – to cache content close to your users and also close to Google. When milliseconds count, the closer the server is to Google, the faster the response time will be. Pugpig uses the Fastly CDN to make sure that your sites and apps perform quickly for your audiences no matter where they are and also for Google and other search engines.

Beyond response time there are three other key metrics to evaluate within the Search Console: response codes, file types and crawl purpose. Response codes show what responses the crawler is getting from the site, too many errors indicate problems that need to be addressed such as hard or soft 404s. File types demonstrate the type of content – HTML articles, images and other files – and for media sites the vast majority should be HTML articles. Crawl purpose shows if the content is new or has been refreshed; anything over 10% “discovery” could indicate a problem with too many URLs.

Once these issues are understood and addressed, there are a few other areas that need to be considered, and this is where it’s best to use an SEO audit tool like Semrush. An SEO audit tool will give a more detailed report which will identify issues that the Search Console doesn’t see, for example duplicate content, overly long pages titles and internal redirects.

SEO is a complex subject but it’s important to ensure that the approach is regularly evaluated and optimised so that content remains discoverable for the user. Pugpig’s Consulting Services team is helping our publishing partners optimise their content and website to improve SEO. If you would like to discuss how we can help you, contact us at info@pugpig.com.

WAN-IFRA’s World Press Trends report shows impact of inflation on publishers

WAN-IFRA has released its annual World Press Trends report, and the newspaper industry group noted a decidedly negative turn in publishers’ outlook compared with last year, going as far as say that it made “sobering reading”. It is a comprehensive survey of news publishers, canvassing 167 news executives in 62 countries, and this year 55.4% said that they were pessimistic, a stark reversal from a year ago when 80% of respondents said that they were optimistic about the next year.

Pressures on their print business, both in terms of declining revenue but also increases in costs, were a drag on results. Print advertising and circulation dropped from 56.1% of respondents income to 53.5%. Print is still a majority of their revenue, but at the same time, print costs rose to 19.6% of the total versus 14.5% of costs a year ago.

However, one thing that the report’s authors noted was that responses were not uniform across markets. The report said that the outlook was largely a “tale of two worlds”, one for developed and another for developing markets. In 2021, publishers in developing markets only expected revenue growth of 8%, but in 2022, the figured jumped to 24% growth year on year. However, the report’s authors found that publishers in developed markets predicted a drop in revenue growth from 12% in 2021 to 8% in 2022. Another major difference between developed and developing markets was publishers’ primary revenue focus. Subscriptions and reader revenue were a much higher focus in developed markets, while digital advertising was a much higher priority in developing markets because subscriptions and memberships haven’t “caught on with readers in those regions”.

With softness in the more traditional sources of revenue, driving publishers across markets are seeking new sources of income, which WAN-IFRA said included events, contract publishing and e-commerce. Earnings from these new sources of revenue grew 21.5% last year, the largest increase of any segment for the newspaper publishers. It echoes a finding in our own State of the Digital Publishing Market report at the end of last year, which found that publishers were seeking their next new source of growth after subscriptions. And next year, news publishers expect that these new sources of revenue will account for 24% of their income. They expect events, e-commerce, partnerships with platforms and memberships to be the fastest growing segment. However, it’s important to note that publishers expect an increase in grant funding as well, particularly in developing countries.

Income is one thing, but where publishers expect to invest also tells a story. While these new sources of revenue are expected to drive growth, investment in them comes third after investments in reader revenue at second and investments in product development and R&D topping the list. Of course, product development can include investments in new sources of revenue. When it comes to technology, 90% of publishers are looking to invest in data analytics and intelligence as well as increasing their multimedia capabilities including video and audio, including podcasts.

Looking into the near future, 69% of publishers expect artificial intelligence to have an impact on their business in the next two to three years, with 5G mobile technology and the metaverse rounding out the top three technologies that would affect the news industry. And the report highlighted that while the current focus about coverage of AI focuses on generative tools like ChatGPT, many publishers and technology providers have been using AI for automation such as natural language processing and semantic analysis that automatically and consistently tags content.

Pugpig Consulting Services has been working with our publishing partners to implement semantic analysis tools that help to manage tags more effectively and efficiently so that it reduces time to manage tags and can eliminate the creation of duplicative tags. This improves the user experience by improving related content discoverability, and reducing and improving tags also supports your SEO strategy by eliminating tag competition.

To access the full report, visit the WAN-IFRA website.

Industry News

Here are some of the most important headlines about the business of news and publishing as well as strategies and tactics in product management, analytics and audience engagement.