Fast site indexing: how to speed up indexing of a large content project

One of the primary goals for a SEO specialist during promotion of a large content project is making the conditions for search engines to index as more pages of a site as possible, and to do it as fast as possible. So we should not just wait for a indexing, but make this process faster.

Terms when you can use indexing speed up methods:

• Your site has more than several thousands pages

• The majority of these pages are similar (profiles in social networks, product pages in online shops, classified ads on advertising sites, news pages etc.)

• The main site navigation already includes more than two levels in URL structure

• Users increasingly attend to search function

• Impossibility to place/buy/exchange loads of backlinks for indexing of each page from loads of pages on your site becomes obvious.

So, what methods will we use for speeding up indexing of your site? Here they are:

• Adding a sitemap to search engines

• Competent internal linking

• Main site navigation

• “Breadcrumbs”

• Introduction to a tagging system

• SEO-trick: an upgraded sitemap

Adding sitemap to search engines

It is the easiest part. Most of search engines support the same Sitemap XML format for a sitemap that can be added to search engines in several ways.

Figure 1. The example of a typical map in XML format

The easiest way is to create a sitemap with special programs and to indicate its presence in the file robots.txt (Google welcomes this method).

A more complicated way but not fundamentally different from the first one is to add files from your search engine account.

You should remember all the time (and of course do it!) that files should be regularly updated according to changes on your site.

Competent internal linking

We will speak about two points in this part:

a) Main site navigation and

b) So-called «breadcrumbs».

The main website navigation is a list of all major site sections repeated from page to page. Thereby, we show search engines the weight (the significance of these pages in the general site hierarchy).

Figure 2. The example of the main site navigation

If a page A has 10 pages linking to it, and page B has 100, it’s quite logical that page B will receive more attention from search engines (and search engines algorithms are based primarily on logic). If all pages of a site link to the main page, search engines robots will visit it more often than the others.

The «breadcrumbs» method has the same idea: to link all site pages between each other.

Hansel and Gretel in the Andersen’s old fairy tale found their way home from the woods with help of bread crumbs which they had left on the way. In our case we will leave «bread crumbs» for search robots.

“Breadcrumbs” is an element of site navigation which allows search robots to maneuver quickly inside a site and to index its content, and also it helps users to understand easily where they are at the moment. With help of “breadcrumbs” you can also easily show search engines the importance of various pages in an overall hierarchy of the site.

Figure 3. The example of “breadcrumbs”

Introduction to a tagging system

Tagging system is another great way to help both users (in search) and search engines (in web pages indexing).


Figure 4. The tags cloud from the site www.inetgiant.com

Tagging is assigning of certain thematic labels to different pages. Tag cloud can be generated automatically, or if to speak for example about ads sites, you can allow free use of tags except stop words (without this point you have a risk to get a stream of spam with “adult ” topics tags).

SEO-trick: an upgraded site map

All methods mentioned above are good, but they: a). are used by everyone; b). may be not fast enough for you when a million of pages are waiting for indexing. So, what to do?

Let’s try to approach the problem from a logical point of view. We made internal linking, put backlinks, but where is a reserve?

Reserve in our case is an economy of server time which search engines robots spend for indexing. Particularly it can be used while site maps generating of special type. There are a lot of such map design variants, but the main goal is to make pages as “light” as possible and to offer them not for indexing but for passing. Have a look how it is realized, for example, on a site 123people.com:

1. Main page

Figure 5. Main page of site map 123people.com

2. Pages of the second, the third and the fourth (if it is necessary) level in the URL structure (in alphabetical order)

Figure 6. The page of the 2nd level of the sitemap 123people.com

3.  All pages of the site contain links to this sitemap

Figure 7. Links to the site map in footer

4. After all we have as simple page code as possible (without using styles, java-scripts or any garbage code reading which takes valuable time of search engine robots).

5. But the main feature is that the following directives need to be present in a code of each page of the sitemap:  > meta name = “robots” content = “noindex, follow, noarchive” / <. That means that robot receives a command: “You can enter the house, explore rooms, but you are not allowed to touch anything.”

I.e., crawler doesn`t index content of this page (noindex), doesn`t cache them (noarchive), but follows links on a page and indexes them much faster.

For example, Google offers such result for a query “site: 123people.com”:

Figure 8. Search results in Google for the query “site: 123people.com”

Conclusion:

There are 3 ways to make indexing faster:

  • Competent internal linking (with help of main navigation, “breadcrumbs” and tagging system)
  • adding a sitemap to search engines
  • creating and competent adjusting of a special HTML sitemap

Before using any listed method you should check the indexing level of a site, learn a number of pages which are in web index of the major search engines, fix this figure and regularly monitor the dynamics. Only in such way you can estimate efficiency of the particular method for your site.

By combining these ways you can speed up indexing of a site of any size and complexity, whether it is a small online store or a social network with millions of profiles.

 

Share on Facebook0Tweet about this on TwitterShare on Google+0Pin on Pinterest0

Recent posts

Top-5 Digital Marketing Experts to Follow

October 13, 2017

We want to devote this post to education and self-development. Our team is ready to share what digital marketing trendsetters we follow to stay updated and get inspired for new projects. No matter whether you own an ecommerce business, manage a marketing team, or work as a digital marketer, you will find something helpful in the Continue reading >

LSI Keywords: Everything You Wanted to Know about Them

October 12, 2017

I haven’t written blog posts for a while. It’s now or never because this evening I’m leaving for Outsource People 2017 in Kyiv. So, the topic I would like to cover today is related to LSI keywords. Do you use them on your website? Because you should. These keywords can help you reach top-10 in Continue reading >

Facebook Canvas: The What, Why, and Who Cares?

October 4, 2017

Are you that person who spends over 4 hours a day on your mobile phone? Some studies show that the U.S. users are spending up to 5 hours every day on their mobile devices. Considering the aforementioned, it is obvious that advertisers are interested in reaching mobile users. Facebook created an interactive post format called Continue reading >

Friday Catch-Up: Banner vs Text Ads

September 29, 2017

Do you know the difference between banner and text ads? It appears that in the world of online advertising this is not so obvious. The choice is getting harder when you want to launch a new PPC campaign. The following explanation is designed to highlight the advantages and disadvantages of banner and text ads to make Continue reading >

Let's get the ball rolling

Please fill in this short form and we will be in touch with you soon

For any questions contact@promodo.com

UK | USA

+44 (0) 20 313 766 81
+44 7852 537715

Lincoln
The Terrace AT5,
Grantham Street,
LN2 1BD

+1 347 809 34 86

Las Vegas
723 S Casino
Center BLVD FL 2
NV 89101

start now

Start now