Dignity nobility top quality point out highness standing course caste status superiority priority manner distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robot concerning which web pages aren't to get crawled. As a search engine crawler may possibly https://www.youtube.com/watch?v=imiGL2YS2uo