Fast site indexing: how to speed up indexing of a large content project

Category: SEO

One of the primary goals for a SEO specialist during promotion of a large content project is making the conditions for search engines to index as more pages of a site as possible, and to do it as fast as possible. So we should not just wait for a indexing, but make this process faster.

Terms when you can use indexing speed up methods:

• Your site has more than several thousands pages

• The majority of these pages are similar (profiles in social networks, product pages in online shops, classified ads on advertising sites, news pages etc.)

• The main site navigation already includes more than two levels in URL structure

• Users increasingly attend to search function

• Impossibility to place/buy/exchange loads of backlinks for indexing of each page from loads of pages on your site becomes obvious.

So, what methods will we use for speeding up indexing of your site? Here they are:

• Adding a sitemap to search engines

• Competent internal linking

• Main site navigation

• “Breadcrumbs”

• Introduction to a tagging system

• SEO-trick: an upgraded sitemap

Adding sitemap to search engines

It is the easiest part. Most of search engines support the same Sitemap XML format for a sitemap that can be added to search engines in several ways.

Figure 1. The example of a typical map in XML format

The easiest way is to create a sitemap with special programs and to indicate its presence in the file robots.txt (Google welcomes this method).

A more complicated way but not fundamentally different from the first one is to add files from your search engine account.

You should remember all the time (and of course do it!) that files should be regularly updated according to changes on your site.

Competent internal linking

We will speak about two points in this part:

a) Main site navigation and

b) So-called «breadcrumbs».

The main website navigation is a list of all major site sections repeated from page to page. Thereby, we show search engines the weight (the significance of these pages in the general site hierarchy).

Figure 2. The example of the main site navigation

If a page A has 10 pages linking to it, and page B has 100, it’s quite logical that page B will receive more attention from search engines (and search engines algorithms are based primarily on logic). If all pages of a site link to the main page, search engines robots will visit it more often than the others.

The «breadcrumbs» method has the same idea: to link all site pages between each other.

Hansel and Gretel in the Andersen’s old fairy tale found their way home from the woods with help of bread crumbs which they had left on the way. In our case we will leave «bread crumbs» for search robots.

“Breadcrumbs” is an element of site navigation which allows search robots to maneuver quickly inside a site and to index its content, and also it helps users to understand easily where they are at the moment. With help of “breadcrumbs” you can also easily show search engines the importance of various pages in an overall hierarchy of the site.

Figure 3. The example of “breadcrumbs”

Introduction to a tagging system

Tagging system is another great way to help both users (in search) and search engines (in web pages indexing).


Figure 4. The tags cloud from the site www.inetgiant.com

Tagging is assigning of certain thematic labels to different pages. Tag cloud can be generated automatically, or if to speak for example about ads sites, you can allow free use of tags except stop words (without this point you have a risk to get a stream of spam with “adult ” topics tags).

SEO-trick: an upgraded site map

All methods mentioned above are good, but they: a). are used by everyone; b). may be not fast enough for you when a million of pages are waiting for indexing. So, what to do?

Let’s try to approach the problem from a logical point of view. We made internal linking, put backlinks, but where is a reserve?

Reserve in our case is an economy of server time which search engines robots spend for indexing. Particularly it can be used while site maps generating of special type. There are a lot of such map design variants, but the main goal is to make pages as “light” as possible and to offer them not for indexing but for passing. Have a look how it is realized, for example, on a site 123people.com:

1. Main page

Figure 5. Main page of site map 123people.com

2. Pages of the second, the third and the fourth (if it is necessary) level in the URL structure (in alphabetical order)

Figure 6. The page of the 2nd level of the sitemap 123people.com

3.  All pages of the site contain links to this sitemap

Figure 7. Links to the site map in footer

4. After all we have as simple page code as possible (without using styles, java-scripts or any garbage code reading which takes valuable time of search engine robots).

5. But the main feature is that the following directives need to be present in a code of each page of the sitemap:  > meta name = “robots” content = “noindex, follow, noarchive” / <. That means that robot receives a command: “You can enter the house, explore rooms, but you are not allowed to touch anything.”

I.e., crawler doesn`t index content of this page (noindex), doesn`t cache them (noarchive), but follows links on a page and indexes them much faster.

For example, Google offers such result for a query “site: 123people.com”:

Figure 8. Search results in Google for the query “site: 123people.com”

Conclusion:

There are 3 ways to make indexing faster:

  • Competent internal linking (with help of main navigation, “breadcrumbs” and tagging system)
  • adding a sitemap to search engines
  • creating and competent adjusting of a special HTML sitemap

Before using any listed method you should check the indexing level of a site, learn a number of pages which are in web index of the major search engines, fix this figure and regularly monitor the dynamics. Only in such way you can estimate efficiency of the particular method for your site.

By combining these ways you can speed up indexing of a site of any size and complexity, whether it is a small online store or a social network with millions of profiles.