User-agent: * Disallow: /ga/ Disallow: /Images/2/ Disallow: /Images/9/ Disallow: /Datasheets/1/ Disallow: /Datasheets/2/ Disallow: /Datasheets/9/ Disallow: /common/footprintPreview.php* User-agent: BLEXBot Disallow: / # No idea what this is but its fetching 19,000 page a day by reading the same pages # over and over. Found 38 fetches of the same page with the same parameters # Asked to leave because of the incompetent server load generation User-agent: Barkrowler Disallow: / # Some academic research thing that crawls the same page many times a day. User-agent: BUbiNG Disallow: /