Originally published in French in May 2019. Read the French version. Screenshots and charts are in French.
At the talk I gave at SEO Campus Paris 2019, I had the chance to cover a subject I am passionate about: log analysis. In the first part, I dug into the factors that influence how a site is crawled, going through the pillars of SEO: content with word count, technical with load time, structure with depth and internal links, and link building with external links.
To carry out the analysis, I ran the experiment on 12 websites of different topics and sizes, to get a panel of URLs representative of small and medium-sized sites. In total I gathered 200,000 URLs and 7,000,000 Googlebot hits (desktop and mobile). As tools, I used Oncrawl, SEOlyzer, Screaming Frog’s Log File Analyser and SEO Tools for Excel.
Contents:
Content: word count
The orange bars show the average number of hits per URL. The grey area shows the estimated traffic of those URLs, based on the rankings and search volume of the keywords each URL ranks for.
Word count per page does influence Googlebot’s crawl. Pages with 2,000 to 3,500 words are hit far more than the others, between 285 and 726 hits on average per URL. By comparison, the other pages are hit between 6 and 72 times on average.
For pages that rank, there is a significant peak in visibility in Google’s results for pages with 500 to 3,000 words, and especially between 1,000 and 2,000 words.
Favour content of around 2,000 words
Technical: load time
For load time, there are more hits on pages loading in 1 to 3 seconds, even if the figures are not as clear-cut as one might have expected. This supports the theory that Google crawls a URL even if it is slow to load, but will crawl fewer pages overall. So load time still deserves as much optimisation as possible.
As for visibility, there is a significant peak on URLs that load in 1 second. That confirms the importance of load time, not to mention its impact on user experience.
Take care of load time, even more for rankings and user experience than for crawling
Structure: depth
Page depth, as you might expect, has a strong impact on Googlebot’s behaviour and on the average number of hits per URL. Home pages are hit 112 times on average, pages at depth 2 are hit 82 times, 60 times at depth 3, and so on, down to 10 hits on average per URL beyond depth 6. URLs that are not in the internal linking at all are hit only 16 times on average.
In terms of organic traffic, the most visible pages are those at depth 1 and 2, with visibility close to zero for deeper pages.
Watch your page depth: do not put your SEO pages more than 3 clicks from the home page
Structure: internal links
Another parameter with a strong influence on crawling is the number of internal links. The average number of hits per URL rises with the number of internal links: the more a URL is linked, the more it is hit, and the figures are really significant on this metric. In the panel studied, a URL receiving no internal link gets 16 hits on average, against up to 363 hits on average for a URL receiving more than 400 internal links.
In terms of organic traffic, it is less clear-cut. The URLs with the most traffic from search engines are those with 1 to 100 internal links, and also those with more than 400. What to remember is that the most heavily linked URLs come out on top on both dimensions.
Make sure your SEO pages are well served by internal linking
Link building: external links
After internal links, let us look at external links. Pages that receive backlinks are hit more than those that do not, especially once the number of backlinks goes above 60. A page with no external link gets 22 hits on average, against 1,179 hits for a page receiving 60 to 70 external links. Link building therefore matters: like internal linking, it increases how often URLs are crawled.
A real link building strategy should be in place for your SEO pages
Organic search: top 10 keywords
Finally, what happens on URLs that already rank and whose SEO has already been worked on? There is a correlation between top 10 rankings and the average frequency of Googlebot hits. A URL with no keyword in Google’s top 10 is hit 20 times on average, while a URL with at least one keyword in the top 10 is hit at least 201 times. And it keeps rising with the number of top 10 rankings, up to 410 hits on average for a URL with more than 50 keywords in the top 10.
The explanation is that Google pays even more attention to sites that rank well, since users are far more likely to click on them than on a site buried in the deeper results pages. It therefore checks regularly that the site is reachable, that the content of the top pages is still the same, and so on. That is encouraging: the more you optimise the SEO of your pages, the more they are hit, and perhaps the better they rank afterwards. So let us pamper our SEO pages.
Logically, there is also more organic traffic on pages with keywords ranking in Google’s top 10, with a clear drop for those with more than 50 keywords in the top 10.
Good organic rankings bring more Googlebot crawling to your site
Key takeaways: the 6 decisive SEO criteria
To optimise the crawl of your SEO pages, leading to better indexing and therefore better rankings:
- Content should be over 2,000 words
- The page should load in 1 second
- Do not go beyond a depth of 3
- Have as many internal links as possible
- Build a link building strategy (more than 60 backlinks)
- Look after your SEO pages to push their rankings even further
It is a virtuous circle: the more you optimise your SEO, the more crawl budget your site gets
