to prevent undesirable material while in the lookup indexes, site owners can instruct spiders not to crawl specific documents or directories throughout the regular robots.txt file in the basis Listing from the domain. https://vng.gr/ypiresies-seo/