User-agent: W3C-checklink Disallow: User-agent: YandexSomething # Russia Disallow: / User-agent: OpenAcoon # www.OpenAcoon.de Disallow: / User-agent: ICC-Crawler # Japan Disallow: / User-agent: ichiro # Japan Disallow: / User-agent: Combine # http://combine.it.lth.se/ Disallow: / User-agent: wwwster # Germany Disallow: / User-agent: 008 # 80legs - annoying crawler with errors Disallow: / User-agent: * # all other robots and crawlers Disallow: /2009/ Disallow: /Benefits to Member Congregations of RI_files/ Disallow: /Congregational pledge_files/ Disallow: /Individual Pledge_files/ Disallow: /July1_files/ Disallow: /Organization pledge_files/ Disallow: /Resources/ Disallow: /_borders/ Disallow: /_derived/ Disallow: /_fpclass/ Disallow: /_overlay/ Disallow: /_private/ Disallow: /_themes/ Disallow: /_vti_bin/ Disallow: /_vti_cnf/ Disallow: /_vti_log/ Disallow: /_vti_pvt/ Disallow: /_vti_txt/ Disallow: /curriculum/ Disallow: /images/ Disallow: /linked_files/ Disallow: /lists/ Disallow: /september october 2007_files/ Disallow: /stdlib/ Disallow: /testing/