| commit | 12abe51a3b028f49f137c3449b99fc320aac0270 | [log] [tgz] |
|---|---|---|
| author | Charles Smith <techdocsmith@gmail.com> | Fri Nov 03 14:37:01 2023 -0700 |
| committer | Charles Smith <techdocsmith@gmail.com> | Fri Nov 03 14:37:01 2023 -0700 |
| tree | 01bbde19228af7af41d6157a0b8446e4f9e6a107 | |
| parent | 9fadbcbc418bf5c840b72fe4b8c209610c057727 [diff] |
add robots.txt to src
diff --git a/static/robots.txt b/static/robots.txt new file mode 100644 index 0000000..c630a83 --- /dev/null +++ b/static/robots.txt
@@ -0,0 +1,16 @@ +# robots.txt for https://druid.apache.org + +# Keep robots from crawling old Druid doc versions + +User-agent: * +Disallow: /docs/0*/ +Disallow: /docs/1*/ +Disallow: /docs/2*/ +Disallow: /docs/3*/ +Disallow: /docs/4*/ +Disallow: /docs/5*/ +Disallow: /docs/6*/ +Disallow: /docs/7*/ +Disallow: /docs/8*/ +Disallow: /docs/9*/ +