Distribution

Distribution is the process of communicating your targeted pages to Google. In this chapter, we will examine the workings of sitemap stacking, notifying and quality assurance. The first and most important step when it comes to distribution is to create a whole site inventory sitemap.

Sitemap Stacking

In its simplest form, stacking means listing all your targeted pages into a single sitemap. A sitemap is an XML file that lists the URLs of targeted pages. It allows us to add additional information about each URL, such as when the content was last changed, and if the page contains embedded images and videos. I won’t go into the finer details of this format, because you can find most of the specifics in the official specification at http://www.sitemaps.org/protocol.html and of course, Google’s spec, [google sitemap.xml].

There are a few things you should keep in mind when sitemap stacking:

• The sitemap should be made available at the root of your site. Keep it simple, so to not run into multiple reading rights issues, and place it in https://www.example.com/sitemap.xml format:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/a/page</loc>
    <lastmod>2017-09-22T14:05:07+00:00</lastmod>
  </url>
</urlset>

• The <loc> is the URL of the targeted pages.
• The <lastmod> is the date when the content was last changed. Use this if your site’s architecture depicts the times that the content of your pages was last changed.
• If your pages contain embedded images or self-hosted videos, you can also add these to the sitemap. In fact, I recommend doing this for everything self-hosted. As videos are mostly Youtube embeds, there is no need to add them in the sitemap.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
  xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
  xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
  <url>
    <loc>https://www.example.com/a/page</loc>
    <image:image>
      <image:loc>https://www.example.com/i/book-writing.jpg</image:loc>
      <image:caption>Writing a book about SEO</image:caption>
    </image:image>
    <video:video>
      <video:content_loc>
        https://www.example.com/book-writing-video.flv
      </video:content_loc>
      <video:player_loc allow_embed="yes" autoplay="ap=1">
        https://www.example.com/videoplayer.swf?video=123
      </video:player_loc>
      <video:thumbnail_loc>
        https://www.example.com/thumbs/book-writing.jpg
      </video:thumbnail_loc>
      <video:title>Writing a book about SEO</video:title>
      <video:description>
        Time-lapse video capturing the course of writing a book about SEO
      </video:description>
    </video:video>
  </url>
</urlset>

• Make sure the sitemap is UTF-8 encoded, and the URLs within the file are entity escaped. It’s a typical rookie mistake to make, even though it’s explicitly stated to do so in the spec at http://www.sitemaps.org/protocol.html#escaping.
• Every targeted page must be stacked in a sitemap. If a targeted page is not listed in a sitemap, it is not a targeted page and should not exist or not get indexed.

There are plenty of other fields that you can add to your sitemap, for example:

• priority
• mobile
• rel-alternate
• changefreq
• image
• video

My recommendation is: Don’t do it! The changefreq and priority fields are simply overhead that make no discernable difference to your site. For the mobile and rel-alternate language fields, it is an easier and testable option to place them into the HTML head.

The rel-alternate mobile and rel-alternate language can be set on three different levels. They are:

• in the sitemap
• via HTTP-Headers
• within HTML Markup (the canonical)

It is important that you choose one from the list above. Do not manage this kind of information at multiple levels. I recommend using HTML markup because it’s much easier to see if the correct metadata is in the markup of the HTML page. Developers naturally look at the HTML markup source of their pages, but much more rarely at the HTTP headers and, sadly, most of the time nobody looks to check if the sitemap is still correct. Therefore, it’s best to keep “invisible” meta-information where people can see it in the HTML.

The simplest stacking logic is to write every targeted page in one sitemap.xml. A graph, such as the one below, will show you that you have, in total, z targeted pages (as every targeted page must be listed in the sitemap) and that x% of these pages have been indexed by Google.

If you have a more complicated site with different page types, I recommend you to stack your targeted pages into separate sitemaps. For example:

• https://www.example.com/n-sitemap.xml
• https://www.example.com/l-sitemap.xml
• https://www.example.com/t-sitemap.xml

The sitemap logic should mirror the namespace logic you have used for your targeted page URLs. With this setup, you will get a different indexing graph for each of your page types.

Now you can talk about communicated and indexed pages for each page type, as opposed to the first more simple setup, where you could only make whole sitewide statements. So now instead of saying, “Some of our pages are not getting indexed!” you can now say, “Pages of type ‘n’ have a lower indexing rate than pages of type ‘l’.” “What is the difference between them?”.

Unfortunately, there is a limitation to sitemaps - they cannot have more than 50,000 page URLs listed. At the point when a sitemap becomes too big, you will need to split them up (with 50,000 page URLs per sitemap). It will then look something like this:

• https://www.example.com/n-sitemap-1.xml
• https://www.example.com/n-sitemap-2.xml
• https://www.example.com/n-sitemap-3.xml
• https://www.example.com/n-sitemap-4.xml

To further achieve the graph that will tell you how many page types have been submitted to Google and how many of them have been indexed, you will need to wrap them in a sitemaps-index.xml file (a simple list that lists simple lists).

• https://www.example.com/n-sitemap-index.xml

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/n-sitemap-1.xml</loc>
    <lastmod>2017-09-22T14:05:07+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>http://www.example.com/n-sitemap-2.xml</loc>
    <lastmod>2017-09-22T14:05:07+00:00</lastmod>
  </sitemap>
</sitemapindex>

Note: Although they look similar, the sitemap-index format is not the same as the sitemap format. Please check the official manual for the difference between these two formats - it’s subtle but important.

With the sitemap-index, you will now get a graph that exceeds 50,000 submitted URLs.

You can, for experimentation reasons, stack URLs in multiple sitemaps (double submission is okay). For example:

• https://www.example.com/all-targeted page-with-less-than-5-items-sitemap.xml
• https://www.example.com/all-targeted page-with-more-than-5-items-sitemap.xml

Doing this will allow you to receive submitted (number of pages) and indexed values for different quality criteria of your site. If you want to narrow down why some kinds of pages may not be getting indexed, this action will be of great value. It’s okay to have these pages in both the namespace-sitemap.xml files as well as in the experimental ones.

Your sitemaps must be updated automatically (if you deleted or added new pages) at least once a day. As a bare minimum, you must complete at least the simple sitemap logic (all your inventory, all your targeted pages). An actively maintained, growing site without a working distribution, is completely and utterly broken.

Sitemap Notifying & robots.txt

After your sitemap has been created, the next step is to notify Google of your sitemap.xml-files. Notifying Google is very simple. All you need to do is:

1. List all your sitemap-index.xml files in the robots.txt.
2. List all sitemap.xml files that are not part of a sitemap-index.xml, directly in the robots.txt.

robot.txt is a text file that will let Google know how to crawl your content. It is simply a file that must be placed at the root of your domain, so for example https://www.example.com/robots.txt.

We are going to discuss this all in more detail over the next few pages, but you should know that with this step, and after Google has crawled the robots.txt (about every 24 hours), you will have communicated your complete inventory to Google. It’s not yet crawled, and, not yet indexed, but at least Google knows your stuff exists.

It’s important to make sure that listing your sitemap.xml in the robots.txt is an automatic process. As soon as a new sitemap.xml exists, it must be listed in the robots.txt. When you do it, it will probably look something like this:

User-Agent: *
Disallow:
Sitemap: https://www.example.com/n-sitemap-index.xml
Sitemap: https://www.example.com/z-sitemap-index.xml
Sitemap: https://www.example.com/a-sitemap.xml
Sitemap: https://www.example.com/d-sitemap.xml

So what’s the meaning of all of this?
“User-Agent: *
Disallow:”
This means nothing is to be dismissed on your page, and the result is that every bot is permitted to crawl everything and anything. The robots.txt is a crawling directive, not an indexing directive. So, whenever you block something, Google might index it anyway. It will not know much about the URL, but it might index it anyway.

In 99% of all cases, you want to manage indexing (which determines if it shows up in Google or not), not crawling. To manage indexing, always use the <meta name="robots" content="noindex"> tag, and never use the robots.txt. If you block something with the robots.txt, Google cannot see the content anymore, so even onpage <meta name="robots" content="noindex"> would be ignored on that page.

Only in 1% of cases you want to control crawling of a certain resource. And even then, I recommend you rethink your internal application logic instead of blocking it with the robots.txt, as you might trade in a crawling issue with an indexing issue.

If you want to block other crawlers because of “reasons” (although I do not recommend it), you can do that, but always make sure that you do not block Googlebot from anything. Malicious crawlers (for example, ones looking for email addresses to spam) will crawl you anyway, regardless of what you write into the robots.txt. If it is important that content is not crawled, I recommend some other method (i.e., HTTP-Auth IP-Blocking) on the server side to prevent this.

Therefore, just do:

User-Agent: *
Disallow:

Every other disallow statement must be seen as a bug and should be solved in another way.

Important note: Never, ever block images, CSS or JS. Google must be able to render the site the same way your users do. If you block JS, CSS or images, your site will not perform. This, therefore, is a direct violation of the first page of the Google Webmaster Guidelines:

Never!!:

User-Agent: *
Disallow: /js/
Disallow: /css/
Disallow: /images/

Private Sitemaps

In certain cases, you may not want to have your sitemap.xml listed in the robots.txt. One of these instances could be that you want Google to crawl you, but you do not want your competition to easily infer how big your page inventory is. My opinion is this rarely makes sense, because it can be circumvented with other techniques. However, there probably is 1% of cases where this may make sense. In the 1% of cases, give your sitemap.xml difficult to guess filenames such as https://www.example.com/a-sitemap-franz-was-here.xml

And don’t notify Google via robots.txt but via Google Search Console (where you can manually submit a sitemap.xml). This way, Google will know of them, too. The downside of this approach is that other search engines don’t know about you (which doesn’t really matter), but also that people tend to forget to submit a completely new sitemap.xml (if not in the next six months, then in five years) and then wonder why Google doesn’t index them properly.

Getting Crawled And Indexed

Ok, so you’ve done everything right. You listed your complete inventory of targeted pages in the sitemap.xml and you notified Google of your sitemap(s).xmls. Google will now crawl you and hopefully, index you.

Crawling is the process of Google accessing your page and downloading its content. Your biggest enemy for this (after you have done the stacking and notifying correctly) is your server performance and arithmetic. Google assigns a certain rate to crawl your site, a so-called “crawl budget.” It’s based on overall content quality, estimated importance and a lot of other stuff we don’t know. You can see and influence your crawl rate at Google Search Console > Site Settings.

As you want to be crawled as thoroughly and as fast as possible, it’s a good idea to set this slider to 10/1 (10 requests per second). If this is not possible in your view of Google Search Console, then don’t worry, simply set it to the maximum for now. If you wait a few days, you’ll be able to raise the slider higher. Keep doing this until you have reached 10/1.

At the same time, however, you must make sure that your servers remain fast. You must still be able to reach the golden goal speed values we discussed previously. If your site becomes too slow because of the massive crawling by Google, buy new servers (or get a better connection between your servers), or solve any other technical issues that hurt your servers’ performance.

Additionally, you also need to care about the “time spent downloading a page,” which can be found in Google Search Console.

This graph must show a stable (not spiky) line, and you must deliver an average value of about < 300ms. The “time spent downloading a page” graph is not about the first byte of the page, but more like the last byte. And, if your site is slow, even if it is only temporary, then Google will reduce crawling your site.

If you do these things correctly, you will at some point get an average “pages crawled per day” of between 400,000 and 860,000.

Also, you should know that Google has something I call “crawl efficiency”. I put this value at 0.7 on a well optimized site. This means that only 70% of the pages crawled within a given timeframe are unique pages, and the rest are duplicate crawls of the same page. And 0.7 is only the value if you do everything right: if you have lots of parameters and deadweight URLs or pages, the efficiency rate can be considerably lower.

Let’s consider this further with an example. If you have a site with ten million pages and your average crawl rate is 400,000, with the average crawl efficiency being 0.7, the time taken to crawl your site is:
10,000,000 / (400,000 * 0.7) = 35.71 days
Therefore, it would take about 36 days until you can be sure that a new or updated page has been crawled.

However, if you have managed to keep the crawl rate high (~800,000), you will have halved that time. But, if you have 20 million pages, or lots of different URL variations, redirects, and parameters, it can be even longer.

So, this means, it takes time. Whenever you update a big portion of your big site it takes Google quite some time to recognize the updates on all of your pages. In general, I recommend that it must not take more than one month with a presumed crawl efficiency of 0.7 to crawl all your pages.

That’s because, if it takes longer than one month to connect the impact between cause (i.e. onpage optimization) and effect (better visibility and traffic) then we (as busy human beings) don’t see any cause and effect at all. This puts a theoretical limit of the maximum pages on your site to 25 million with best case values. If you have more pages than that, I recommend you split your pages into different websites with different domains.

You can also choose to not set a manual crawl rate in the Google Search Console settings. In this case, Google will assign a crawl rate to you, and this crawl rate might be higher than ten requests per second.

That said, the only cases where this happens are when you:

• a) running an industry leading site with an unbelievably fast server
• b) are hosted at a data center that has a direct glass fiber connection to Google and you have a fast server
• c) have such an unbelievably blazing fast server and a lot of high quality content.

For small sites, mulling over crawl performance and the settings recommended here hardly matter. A higher priority is making sure your servers are stable and the average time spent downloading a page is about 300 ms.

Indexing is Google’s process to store your content in a way that it can be matched with the user’s query. What share of our targeted pages is being indexed depends on mostly unknown factors. What we do know is that the killers for indexing are:

• slow sites
• internal duplicate content
• external duplicate content
• a poor QC
• targeted pages that are not interlinked.

If you have done everything as outlined in my targeted page best practice section, you should expect:

• close to 100% indexing for all targeted pages with unique content
• close to 90% indexing for all targeted pages with a unique combination of content as the main content.

If you are below these benchmark numbers, then you are underperforming. If less than 70% of your targeted pages are indexed, your site must be considered completely and utterly broken. If you have such a case, the reason usually falls into one of these categories:

• speed: the pages do not fulfill the web performance goals
• content: internal or external duplicate content, or no content
• interlinking: pages do not fulfill the interlinking requirements
• other: pages do not systematically fulfill one ot more aspects of the targeted page requirements

Now another important note: Beware of the opening scissors!

Let me explain. If you are constantly creating and submitting more and more targeted pages to Google, you must make sure that your overall indexing rate stays at least the same, or improves.

If you are creating and submitting more pages but the indexing ratio is going down, or you submit a constant number of pages, but the number of indexed pages goes down, the “scissors are widening” as seen in the chart below (Google Search Console > Crawl > Sitemaps > Web Pages). This means something is wrong.

When you notice this happening, I recommend you stop submitting new pages and wait one week. If the scissors begin to close again (the submitted and indexed lines converge again), then you simply have a delay (which can be fixed by increasing your crawl rate).

If the gap doesn’t close, you must immediately check your speed, server, distribution and targeted pages, because one or more of these factors is hurting your webproperty. Remember, if your hard work does not get indexed, then your hard work will not contribute to your traffic and is a wasted opportunity.

Updating Sitemaps

The entire site inventory (your sitemap.xml) should be updated at least once a day. There are many cases, though, when this is not fast enough. One simple method you can use to speed up the indexing of new pages is to create a feed of new and updated material, published as an update.rss.

The update.rss is nothing other than a simple sitemap in the RSS format where you stack list all of your targeted page URLs that were created or updated in the last few hours. Then, if the update.rss has changed (so, if there are pages created or updated in the last few hours) you ping this sitemap.rss to Google. Unlike your main sitemap, this feed shows only new and changed content. To do this, you need to create an automated process that:

• (re)creates https://www.example.com/update.rss (must be in the root of your domain) with new and updated pages every hour
• makes a server side GET-Request to https://www.google.com/webmasters/tools/ping?sitemap=%your url encoded url of the update.rss% if pages were created or updated in the last few hours.

There’s more about sitemap.xml pings at https://en.wikipedia.org/wiki/Sitemaps#Search_engine_submission. If you are in the news vertical, please google [Google News xml].

Distribution Quality Assurance

A website with a broken distribution is a broken website. And rather than having direct feedback like you get when your pages are broken, you do not have this kind of feedback when your distribution breaks. Instead, your traffic will not plunge immediately, but your growth will slowly decline over time.

To prevent this from happening, quality assurance in the form of regular manual checks of the sitemap, and examinations of the key metrics (such as the average time spent downloading a page and pages crawled per day, both available via Google Search Console), is essential.

Automated tests of the sitemap do not hurt either, so make a habit of checking if the update.rss really is being updated in real time, and if the update.rss is still being pinged, and so on.

The Startpage, Targeting, And Linking

It is important to realize that SEO does not care too much about your startpage. From an SEO perspective, the startpage is:

• a targeted page for the brand of your company
• a targeted page for one competitive targeted phrase
• a value hub.

However, it’s important for your brand and the competitive targeted phrase to make sure that your title has a format like this:

<title>%your brand% %targeted phrase%</title>

Start with your brand, and then target something you want to be found for. I recommend choosing one targeted phrase that stands for what your business is. Here are some made up ones so you can get an idea:

<title>CarIns Car insurances online</title>
<title>Crocodile Rock Bungalows on Koh Chang</title>
<title>Pills & Fun Online Pharmacy</title>

It’s essential that your brand name is visible in the text on your startpage as well, and not only as an image. Make sure your competitive targeted phrase is also available in another visible way on the startpage, but keep in mind there is no need to repeat the targeted phrase in all targeting elements on the startpage.

When it comes to linking, you will find that almost 90% of the inlinks from other websites to your webproperty will link to the root of the domain (a link direct to https://www.example.com for instance). This means you must do two things:

1. Your startpage must be in the root of your domain. Google will always see the root of your domain as the location of your startpage, and you should show real content immediately. Do not redirect away from your startpage:
   • Good: https://www.example.com/ -> HTTP 200 must show the content of your startpage
   • Bad: https://www.example.com/ -> redirect (whether HTTP 301 permanent (or HTTP 302 temporary) redirect to https://www.example.com/home.html (or something else).
2. You must link like crazy from your startpage to your targeted pages. To achieve this, I recommend that you have a sensemaking link section on your startpage where you statically link to your most important targeted pages. Additionally, you should have a section that links to your newest and/or latest updates targeted pages. And, of course, a section that serendipitously links to some targeted pages.

In addition to these two things, you should focus on speed and the user experience, but generally, you can do whatever you like on the startpage. There is no need for diversity of content or anything like that here.