# Пример /robots.txt file # # User-agent: webcrawler # Disallow: # # User-agent: lycra # Disallow: / User-agent: * Disallow: /graf/ Disallow: /arxiv/