Fast site indexing: how to speed up indexing of a large content project

One of the primary goals for a SEO specialist during promotion of a large content project is making the conditions for search engines to index as more pages of a site as possible, and to do it as fast as possible. So we should not just wait for a indexing, but make this process faster.

Terms when you can use indexing speed up methods:

• Your site has more than several thousands pages

• The majority of these pages are similar (profiles in social networks, product pages in online shops, classified ads on advertising sites, news pages etc.)

• The main site navigation already includes more than two levels in URL structure

Get lost in the search results?

Let Promodo experts help your website index better by optimising your pages, improving your content, and enhancing your linking strategy.

By clicking this button you agree to our terms & conditions

• Users increasingly attend to search function

• Impossibility to place/buy/exchange loads of backlinks for indexing of each page from loads of pages on your site becomes obvious.

So, what methods will we use for speeding up indexing of your site? Here they are:

• Adding a sitemap to search engines

• Competent internal linking

• Main site navigation

• “Breadcrumbs”

• Introduction to a tagging system

• SEO-trick: an upgraded sitemap

Adding sitemap to search engines

It is the easiest part. Most of search engines support the same Sitemap XML format for a sitemap that can be added to search engines in several ways.

Figure 1. The example of a typical map in XML format

The easiest way is to create a sitemap with special programs and to indicate its presence in the file robots.txt (Google welcomes this method).

A more complicated way but not fundamentally different from the first one is to add files from your search engine account.

You should remember all the time (and of course do it!) that files should be regularly updated according to changes on your site.

Competent internal linking

We will speak about two points in this part:

a) Main site navigation and

b) So-called «breadcrumbs».

The main website navigation is a list of all major site sections repeated from page to page. Thereby, we show search engines the weight (the significance of these pages in the general site hierarchy).

Figure 2. The example of the main site navigation

If a page A has 10 pages linking to it, and page B has 100, it’s quite logical that page B will receive more attention from search engines (and search engines algorithms are based primarily on logic). If all pages of a site link to the main page, search engines robots will visit it more often than the others.

The «breadcrumbs» method has the same idea: to link all site pages between each other.

Hansel and Gretel in the Andersen’s old fairy tale found their way home from the woods with help of bread crumbs which they had left on the way. In our case we will leave «bread crumbs» for search robots.

“Breadcrumbs” is an element of site navigation which allows search robots to maneuver quickly inside a site and to index its content, and also it helps users to understand easily where they are at the moment. With help of “breadcrumbs” you can also easily show search engines the importance of various pages in an overall hierarchy of the site.

Figure 3. The example of “breadcrumbs”

Introduction to a tagging system

Tagging system is another great way to help both users (in search) and search engines (in web pages indexing).


Figure 4. The tags cloud from the site www.inetgiant.com

Tagging is assigning of certain thematic labels to different pages. Tag cloud can be generated automatically, or if to speak for example about ads sites, you can allow free use of tags except stop words (without this point you have a risk to get a stream of spam with “adult ” topics tags).

SEO-trick: an upgraded site map

All methods mentioned above are good, but they: a). are used by everyone; b). may be not fast enough for you when a million of pages are waiting for indexing. So, what to do?

Let’s try to approach the problem from a logical point of view. We made internal linking, put backlinks, but where is a reserve?

Reserve in our case is an economy of server time which search engines robots spend for indexing. Particularly it can be used while site maps generating of special type. There are a lot of such map design variants, but the main goal is to make pages as “light” as possible and to offer them not for indexing but for passing. Have a look how it is realized, for example, on a site 123people.com:

1. Main page

Figure 5. Main page of site map 123people.com

2. Pages of the second, the third and the fourth (if it is necessary) level in the URL structure (in alphabetical order)

Figure 6. The page of the 2nd level of the sitemap 123people.com

3.  All pages of the site contain links to this sitemap

Figure 7. Links to the site map in footer

4. After all we have as simple page code as possible (without using styles, java-scripts or any garbage code reading which takes valuable time of search engine robots).

5. But the main feature is that the following directives need to be present in a code of each page of the sitemap:  > meta name = “robots” content = “noindex, follow, noarchive” / <. That means that robot receives a command: “You can enter the house, explore rooms, but you are not allowed to touch anything.”

I.e., crawler doesn`t index content of this page (noindex), doesn`t cache them (noarchive), but follows links on a page and indexes them much faster.

For example, Google offers such result for a query “site: 123people.com”:

Figure 8. Search results in Google for the query “site: 123people.com”

Conclusion:

There are 3 ways to make indexing faster:

  • Competent internal linking (with help of main navigation, “breadcrumbs” and tagging system)
  • adding a sitemap to search engines
  • creating and competent adjusting of a special HTML sitemap

Before using any listed method you should check the indexing level of a site, learn a number of pages which are in web index of the major search engines, fix this figure and regularly monitor the dynamics. Only in such way you can estimate efficiency of the particular method for your site.

By combining these ways you can speed up indexing of a site of any size and complexity, whether it is a small online store or a social network with millions of profiles.

 

Leave a Reply

avatar
  Subscribe  
Notify of

Recent posts

Call tracking with dynamic number insertion (DNI): how to track the ad source of inbound calls

March 12, 2021

When it comes to effective distribution of marketing budgets, eCommerce businesses that accept orders not only through the digital carts on the website but also through inbound calls, all face the same problem: the inability to see a unified picture of marketing channels that brought customers to their decision to purchase. Here call tracking tools Continue reading >

How to combat cart abandonment with UX: detecting common checkout mistakes of eCommerce brands

February 23, 2021

Online checkout stats shows that more than 80% of online customers abandon their shopping carts. Out of them, about 87% are going to finish their order later. We are not sure if businesses are ready to sacrifice and lose time waiting for those 87% to return and complete the checkout, that’s why in this article Continue reading >

Making the most of email marketing in the beauty industry | Case Study

January 29, 2021

How to engage your mailing subscribers to visit your website? What’s the optimal mailing frequency? How to increase your conversion rates with email marketing? These questions are frequently asked by marketers. The situation gets even more challenging when it comes to the subscribers of a well-known beauty brand. In this material, we share the email Continue reading >

How to market furniture: email, content and remarketing as parts of IKEA marketing strategy

January 28, 2021

Furniture marketing is peculiar for certain reasons. First is that for customers buying furniture this is often emotional and exciting. Second is that this process, when online can take time – there is no way to literally “try on” or test an item. And third is that quality furniture costs money, almost like a small Continue reading >

Let's get the ball rolling

Please fill in this short form and we will be in touch with you soon

For any questions [email protected]

UK | USA | EST

+44 (0) 20 313 766 81
+44 7852 537715

Lincoln
The Terrace AT5,
Grantham Street,
LN2 1BD

+1 347 809 34 86

Las Vegas
6920 S. CIMARRON RD.,
Suite 100,
NV 89113

Tallinn
Roosikrantsi 2-K230,
Kesklinna linnaosa,
Harju maakond,
Tallinn 10119

CEE Ecommerce Report 2019

Based on the analytics data of 292 websites

enter correct name, please
enter correct e-mail, please
SEO

Fast site indexing: how to speed up indexing of a large content project

0
start now

Start now