public-mirrors/NewsBlur-viq
1
0
Fork 0
mirror of https://github.com/viq/NewsBlur.git synced 2025-09-18 21:43:31 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Adding robots.txt so only the homepage will be crawled.

Browse source
This commit is contained in:
Samuel Clay 2011-01-04 08:56:25 -05:00
parent 2ce15eaeee
commit 9cc610cc12
1 changed files with 8 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
media/robots.txt Normal file
View file

@ -0,0 +1,8 @@
User-agent: *
Disallow: /reader
Disallow: /rss_feeds
Disallow: /classifier
Disallow: /analyzer
Disallow: /profile
Disallow: /import
Disallow: /media