Robots Exclusion Standard
When primitive robots were first created, some of them would crash servers by requesting too many pages too quickly. A robots exclusion standard was crafted to allow you to tell any robot (or all of them) that you do not want some of your pages indexed or that you do not want your links followed.


Interviews
SEO Community