Fast site indexing: how to speed up indexing of a large content project

One of the primary goals for a SEO specialist during promotion of a large content project is making the conditions for search engines to index as more pages of a site as possible, and to do it as fast as possible. So we should not just wait for a indexing, but make this process faster.

Terms when you can use indexing speed up methods:

• Your site has more than several thousands pages

• The majority of these pages are similar (profiles in social networks, product pages in online shops, classified ads on advertising sites, news pages etc.)

• The main site navigation already includes more than two levels in URL structure

Get lost in the search results?

Let Promodo experts help your website index better by optimising your pages, improving your content, and enhancing your linking strategy.

By clicking this button you agree to our terms & conditions

• Users increasingly attend to search function

• Impossibility to place/buy/exchange loads of backlinks for indexing of each page from loads of pages on your site becomes obvious.

So, what methods will we use for speeding up indexing of your site? Here they are:

• Adding a sitemap to search engines

• Competent internal linking

• Main site navigation

• “Breadcrumbs”

• Introduction to a tagging system

• SEO-trick: an upgraded sitemap

Adding sitemap to search engines

It is the easiest part. Most of search engines support the same Sitemap XML format for a sitemap that can be added to search engines in several ways.

Figure 1. The example of a typical map in XML format

The easiest way is to create a sitemap with special programs and to indicate its presence in the file robots.txt (Google welcomes this method).

A more complicated way but not fundamentally different from the first one is to add files from your search engine account.

You should remember all the time (and of course do it!) that files should be regularly updated according to changes on your site.

Competent internal linking

We will speak about two points in this part:

a) Main site navigation and

b) So-called «breadcrumbs».

The main website navigation is a list of all major site sections repeated from page to page. Thereby, we show search engines the weight (the significance of these pages in the general site hierarchy).

Figure 2. The example of the main site navigation

If a page A has 10 pages linking to it, and page B has 100, it’s quite logical that page B will receive more attention from search engines (and search engines algorithms are based primarily on logic). If all pages of a site link to the main page, search engines robots will visit it more often than the others.

The «breadcrumbs» method has the same idea: to link all site pages between each other.

Hansel and Gretel in the Andersen’s old fairy tale found their way home from the woods with help of bread crumbs which they had left on the way. In our case we will leave «bread crumbs» for search robots.

“Breadcrumbs” is an element of site navigation which allows search robots to maneuver quickly inside a site and to index its content, and also it helps users to understand easily where they are at the moment. With help of “breadcrumbs” you can also easily show search engines the importance of various pages in an overall hierarchy of the site.

Figure 3. The example of “breadcrumbs”

Introduction to a tagging system

Tagging system is another great way to help both users (in search) and search engines (in web pages indexing).


Figure 4. The tags cloud from the site www.inetgiant.com

Tagging is assigning of certain thematic labels to different pages. Tag cloud can be generated automatically, or if to speak for example about ads sites, you can allow free use of tags except stop words (without this point you have a risk to get a stream of spam with “adult ” topics tags).

SEO-trick: an upgraded site map

All methods mentioned above are good, but they: a). are used by everyone; b). may be not fast enough for you when a million of pages are waiting for indexing. So, what to do?

Let’s try to approach the problem from a logical point of view. We made internal linking, put backlinks, but where is a reserve?

Reserve in our case is an economy of server time which search engines robots spend for indexing. Particularly it can be used while site maps generating of special type. There are a lot of such map design variants, but the main goal is to make pages as “light” as possible and to offer them not for indexing but for passing. Have a look how it is realized, for example, on a site 123people.com:

1. Main page

Figure 5. Main page of site map 123people.com

2. Pages of the second, the third and the fourth (if it is necessary) level in the URL structure (in alphabetical order)

Figure 6. The page of the 2nd level of the sitemap 123people.com

3.  All pages of the site contain links to this sitemap

Figure 7. Links to the site map in footer

4. After all we have as simple page code as possible (without using styles, java-scripts or any garbage code reading which takes valuable time of search engine robots).

5. But the main feature is that the following directives need to be present in a code of each page of the sitemap:  > meta name = “robots” content = “noindex, follow, noarchive” / <. That means that robot receives a command: “You can enter the house, explore rooms, but you are not allowed to touch anything.”

I.e., crawler doesn`t index content of this page (noindex), doesn`t cache them (noarchive), but follows links on a page and indexes them much faster.

For example, Google offers such result for a query “site: 123people.com”:

Figure 8. Search results in Google for the query “site: 123people.com”

Conclusion:

There are 3 ways to make indexing faster:

  • Competent internal linking (with help of main navigation, “breadcrumbs” and tagging system)
  • adding a sitemap to search engines
  • creating and competent adjusting of a special HTML sitemap

Before using any listed method you should check the indexing level of a site, learn a number of pages which are in web index of the major search engines, fix this figure and regularly monitor the dynamics. Only in such way you can estimate efficiency of the particular method for your site.

By combining these ways you can speed up indexing of a site of any size and complexity, whether it is a small online store or a social network with millions of profiles.

 

Leave a Reply

avatar
  Subscribe  
Notify of

Recent posts

SEO Case Study: How to get a niche business in the top of the search results using relevant high-frequency queries

January 20, 2021

Promoting a niche business can sometimes be a challenge. When deploying your forces on the SEO battlefield, it’s extremely important not only to conduct an in-depth competitor analysis but carefully define the requests that are the target in your particular niche. Then come internal and external optimisation, and a number of actions required to improve Continue reading >

How to quadruple website traffic in the real estate niche in 8 months | Case Study

December 21, 2020

Marketing in the real estate niche has certain peculiarities. In this article, we share the case of Kampas, a Lithuanian online business and explain how SEO assets can help to increase visibility in the search results and quadruple the organic traffic! Check the whole story of cooperation on our website. Client Kampas.lt is a real Continue reading >

The future is here: a detailed overview of Google Analytics 4

December 18, 2020

The year 2020, among other things, has brought to the world the long-awaited Google Analytics 4, which is now officially released and available worldwide. What are the key changes and how Google Analytics 4 differs from Universal Analytics? Who will benefit from the transition to the new data model and how to get started with Continue reading >

How to enter a new market using PPC advertising | Case Study

December 1, 2020

How to enter a new market? This is the question lots of retailers who successfully promoted their products within certain regions frequently ask. Marketers tend to believe that display advertising is the easiest way to market a product when entering a new market. In this article, we share the case of an international player and Continue reading >

Let's get the ball rolling

Please fill in this short form and we will be in touch with you soon

For any questions [email protected]

UK | USA | EST

+44 (0) 20 313 766 81
+44 7852 537715

Lincoln
The Terrace AT5,
Grantham Street,
LN2 1BD

+1 347 809 34 86

Las Vegas
6920 S. CIMARRON RD.,
Suite 100,
NV 89113

Tallinn
Roosikrantsi 2-K230,
Kesklinna linnaosa,
Harju maakond,
Tallinn 10119

CEE Ecommerce Report 2019

Based on the analytics data of 292 websites

enter correct name, please
enter correct e-mail, please
SEO

Fast site indexing: how to speed up indexing of a large content project

0
start now

Start now