Right robots.txt

File robots.txt is the main file that describes the processing rules pages of the search engines. This file is needed to specify the primary site name, site map (sitemap.xml), public and private sections of the site.
File robots.txt include s the following directives:.en.pug

• User-agent Directive specifies to what robot the following rules
• * - all robots
• Yandex — main robot Yandex
• Googlebot — Googlebot is the main
• StackRambler — the search engine Rambler
• Aport — Aport search engine
• Slurp is Yahoo's robot
• MSNBot is the MSN robot
• Disallow Directive is the ban part of the site
• Allow — Directive permits part of the website
• Host — Directive specify the primary site name
• Sitemap— Directive guidance a site map (sitemap.xml)
• Crawl-delay Directive specifies how many seconds the robot can wait for a response from the site (required for heavily loaded resources, so the robot did not consider the site is unavailable)
• Clean-param Directive describing the dynamic parameters do not affect the content of the website

Помимо директив в robots.txt используются спец символы:

• * - lubi (including empty) sequence of characters
• $ — is a limitation of the rules

To compile robots.txt use the above guidelines and sung by the characters as follows:

• Specifies the name of the robot on which is written a list of rules
  (User-agent: * - the rule for all robots)
• Spell a list of prohibited sections of the site for the specified robot
  ( Disallow: / - prevent indexing of the entire site)
• Spell a list of permitted sections of the site
  (Allow: /home/ — partition home permitted)
• Specifies the name of the website
  (Host: crazysquirrel.ru — the main name of the site crazysquirrel.ru)
• Specifies the absolute path to the file sitemap.xml
  (Sitemap: https:// crazysquirrel.ru/sitemap.xml)

If the site has no illicit topics, robots.txt must be at least 4 lines:

Check robots.txt and how does it affect indexing of the website by using tools Yandex