Fast site indexing: how to speed up indexing of a large content project

One of the primary goals for a SEO specialist during promotion of a large content project is making the conditions for search engines to index as more pages of a site as possible, and to do it as fast as possible. So we should not just wait for a indexing, but make this process faster.

Terms when you can use indexing speed up methods:

• Your site has more than several thousands pages

• The majority of these pages are similar (profiles in social networks, product pages in online shops, classified ads on advertising sites, news pages etc.)

• The main site navigation already includes more than two levels in URL structure

• Users increasingly attend to search function

• Impossibility to place/buy/exchange loads of backlinks for indexing of each page from loads of pages on your site becomes obvious.

So, what methods will we use for speeding up indexing of your site? Here they are:

• Adding a sitemap to search engines

• Competent internal linking

• Main site navigation

• “Breadcrumbs”

• Introduction to a tagging system

• SEO-trick: an upgraded sitemap

Adding sitemap to search engines

It is the easiest part. Most of search engines support the same Sitemap XML format for a sitemap that can be added to search engines in several ways.

Figure 1. The example of a typical map in XML format

The easiest way is to create a sitemap with special programs and to indicate its presence in the file robots.txt (Google welcomes this method).

A more complicated way but not fundamentally different from the first one is to add files from your search engine account.

You should remember all the time (and of course do it!) that files should be regularly updated according to changes on your site.

Competent internal linking

We will speak about two points in this part:

a) Main site navigation and

b) So-called «breadcrumbs».

The main website navigation is a list of all major site sections repeated from page to page. Thereby, we show search engines the weight (the significance of these pages in the general site hierarchy).

Figure 2. The example of the main site navigation

If a page A has 10 pages linking to it, and page B has 100, it’s quite logical that page B will receive more attention from search engines (and search engines algorithms are based primarily on logic). If all pages of a site link to the main page, search engines robots will visit it more often than the others.

The «breadcrumbs» method has the same idea: to link all site pages between each other.

Hansel and Gretel in the Andersen’s old fairy tale found their way home from the woods with help of bread crumbs which they had left on the way. In our case we will leave «bread crumbs» for search robots.

“Breadcrumbs” is an element of site navigation which allows search robots to maneuver quickly inside a site and to index its content, and also it helps users to understand easily where they are at the moment. With help of “breadcrumbs” you can also easily show search engines the importance of various pages in an overall hierarchy of the site.

Figure 3. The example of “breadcrumbs”

Introduction to a tagging system

Tagging system is another great way to help both users (in search) and search engines (in web pages indexing).


Figure 4. The tags cloud from the site www.inetgiant.com

Tagging is assigning of certain thematic labels to different pages. Tag cloud can be generated automatically, or if to speak for example about ads sites, you can allow free use of tags except stop words (without this point you have a risk to get a stream of spam with “adult ” topics tags).

SEO-trick: an upgraded site map

All methods mentioned above are good, but they: a). are used by everyone; b). may be not fast enough for you when a million of pages are waiting for indexing. So, what to do?

Let’s try to approach the problem from a logical point of view. We made internal linking, put backlinks, but where is a reserve?

Reserve in our case is an economy of server time which search engines robots spend for indexing. Particularly it can be used while site maps generating of special type. There are a lot of such map design variants, but the main goal is to make pages as “light” as possible and to offer them not for indexing but for passing. Have a look how it is realized, for example, on a site 123people.com:

1. Main page

Figure 5. Main page of site map 123people.com

2. Pages of the second, the third and the fourth (if it is necessary) level in the URL structure (in alphabetical order)

Figure 6. The page of the 2nd level of the sitemap 123people.com

3.  All pages of the site contain links to this sitemap

Figure 7. Links to the site map in footer

4. After all we have as simple page code as possible (without using styles, java-scripts or any garbage code reading which takes valuable time of search engine robots).

5. But the main feature is that the following directives need to be present in a code of each page of the sitemap:  > meta name = “robots” content = “noindex, follow, noarchive” / <. That means that robot receives a command: “You can enter the house, explore rooms, but you are not allowed to touch anything.”

I.e., crawler doesn`t index content of this page (noindex), doesn`t cache them (noarchive), but follows links on a page and indexes them much faster.

For example, Google offers such result for a query “site: 123people.com”:

Figure 8. Search results in Google for the query “site: 123people.com”

Conclusion:

There are 3 ways to make indexing faster:

  • Competent internal linking (with help of main navigation, “breadcrumbs” and tagging system)
  • adding a sitemap to search engines
  • creating and competent adjusting of a special HTML sitemap

Before using any listed method you should check the indexing level of a site, learn a number of pages which are in web index of the major search engines, fix this figure and regularly monitor the dynamics. Only in such way you can estimate efficiency of the particular method for your site.

By combining these ways you can speed up indexing of a site of any size and complexity, whether it is a small online store or a social network with millions of profiles.

 

Share on Facebook0Tweet about this on Twitter0Share on Google+0Pin on Pinterest0

Recent posts

SONY DSC

March Round-Up: Recent News from the World of Digital Marketing

April 6, 2016

Finally the warm days have come and the post-Easter mood is in the air. We would like to expand your good mood on our first spring digest, that’s why we prepared a portion of the most recent and talked about news. Follow us and please, do not hesitate to drop us a line in case Continue reading >

Facebook cover

Facebook Audience Targeting Secrets to Lower the Ad Cost

March 30, 2016

Are you a beginner in Facebook ads and need help in effective, low cost ads launch? Then this article is worth reading as it covers the most important issues influencing ad results and expenditures. Moreover, it includes tips, advices and guidance on how to set up audiences.  The two important starting points that are frequently ignored: Continue reading >

keyword search

How Proper Keyword Optimization Can Help Your Business Grow

March 28, 2016

Keyword optimization helps your site appear more frequently in a Google search results and makes it easier for customers to find you.  Even though, search engines are changing their approaches to the website rankings, page keyword optimization still remains an effective method to attract a long-term traffic to your site.  Since your business’ success depends Continue reading >

cover magento

Meet Magento CZ 2016: March 31 – April 1, 2016 – Prague, Czech Republic

March 23, 2016

Little time is left before the main eCommerce event of this spring will take place. Do you have a business of your own, work as a developer or sell online? Take your chance and join us at the first-ever Meet Magento Czech Republic! On March 31, 2016 the wonderful city of Prague will welcome participants Continue reading >

Show all posts