diff --git a/content/docs/api.html b/content/docs/api.html
index ae80fe5..afc029e 100644
--- a/content/docs/api.html
+++ b/content/docs/api.html
@@ -64,16 +64,19 @@
     "tid": "06b318af97ca96c115e878c14d0814a53407751c31388410421c1751@1441467256@<dev.any23.apache.org>",
     "list_raw": "<dev.any23.apache.org>"
 }
+
+Note: date and epoch are in UTC
+
 </pre>
 
 <h3 id='fetchinglistdata'>Fetching list data<a href='#fetchinglistdata' style='color: rgba(0,0,0,0);'>&para;</a></h3>
 <p>Usage:
-<code>GET /api/stats.lua?list=$list&amp;domain=$domain[&amp;d=$timespan][&amp;q=$query][&amp;header_from=$from][&amp;header_to=$to][&amp;header_subject=$subject][&amp;header_body=$body][&amp;quick][&amp;emailsOnly][&amp;s=$s&amp;e=$e]</code></p>
+<code>GET /api/stats.lua?list=$list&amp;domain=$domain[&amp;d=$timespan][&amp;q=$query][&amp;header_from=$from][&amp;header_to=$to][&amp;header_subject=$subject][&amp;header_body=$body][&amp;quick][&amp;emailsOnly][&amp;s=$s&amp;e=$e][&amp;since=$since][&amp;dfrom=$dfrom&amp;dto=$dto]</code></p>
 <p>See below for details of <a href="#Timespans">timespan</a> values</p>
 <p>Parameters:</p>
 <pre><code>- $list: The list prefix (e.g. `dev`). Wildcards may be used
 - $domain: The list domain (e.g. `httpd.apache.org`). Wildcards may be used
-- $timespan: A timespan value (see below)
+- $timespan: A [timespan](#Timespans) value
 - $s: yyyy-mm start of month (day 1)
 - $e: yyyy-mm end of month (last day)
 - $query: A search query (may contain wildcards or negations):
@@ -84,7 +87,13 @@
 - $to: Optional To: address
 - $subject: Optional Subject: line
 - $body: Optional body text
-- quick: send statistics only (exclude participants, threadstruct, word-cloud, emails apart from epoch)
+- $since: number of seconds since the epoch, defaults to now. 
+   Returns '{"changed":false}' if no emails are later than epoch, otherwise proceeds with normal search
+- $dfrom: days ago to start
+- $dto: total days to match
+</code></pre>
+<p>Options:</p>
+<pre><code>- quick: send statistics only (exclude participants, threadstruct, word-cloud, emails apart from epoch)
 - emailsOnly: return email summaries only (omit thread_struct, top 10 participants and word-cloud)
 </code></pre>
 <p>Response example:</p>
@@ -120,7 +129,35 @@
     "name": "dev",
     "cloud": {...},
     "hits": 25,
-    "thread_struct": {...},
+    thread_struct":
+    {
+        "nest": 2,
+        "children": {
+            {
+                "children": {
+                    {
+                        "children": {
+                            {
+                                "children": { },
+                                epoch: ...,
+                                tid: ...,
+                                nest: 1
+                            }
+                        },
+                        epoch: ...,
+                        tid: ...,
+                        nest: 2
+                    }
+                },
+                "epoch": 1474883100,
+                "tid": "b1d6446f5cc8f4846454cbabc48ddb08afbb601a77169f8e32e34102@<dev.ponymail.apache.org>",
+                "nest": 2
+            }
+        },
+        epoch: ...,
+        tid: ...,
+        body: ...
+    },
     "max": 5000,
     "searchlist": "<dev.ponymail.info>",
     "list": "dev@ponymail.info",
@@ -186,13 +223,16 @@
 
 <h3 id='fetchingnotificationsforaloggedinuser'>Fetching notifications for a logged in user<a href='#fetchingnotificationsforaloggedinuser' style='color: rgba(0,0,0,0);'>&para;</a></h3>
 <p>Usage:
-<code>GET /api/notifications.lua</code></p>
-<p>Parameters: <code>None</code> (cookie required)</p>
+<code>GET /api/notifications.lua[?seen=$mid]</code></p>
+<p>Parameters: (cookie required)
+  - $mid: id of the message to be marked as having been seen</p>
 <p>Response example:</p>
 <pre>
 {
     "notifications": {...}
 }
+or
+{"marked": true}
 </pre>
 
 <h3 id='fetchingamonthsdataasanmboxfile'>Fetching a month's data as an mbox file<a href='#fetchingamonthsdataasanmboxfile' style='color: rgba(0,0,0,0);'>&para;</a></h3>
@@ -202,6 +242,20 @@
 <pre>
 TBA
 </pre>
+
+<h3 id='getatomdataforlistoremail'>Get ATOM data for list or email<a href='#getatomdataforlistoremail' style='color: rgba(0,0,0,0);'>&para;</a></h3>
+<p>Usage:
+<code>GET /api/atom/lua(?list=$lid|?mid=$mid)</code></p>
+<p>Parameters: (cookie may be required)
+  - $lid: the list id, e.g. dev@ponymail.apache.org
+  - $mid: The email ID (Permalink)</p>
+<p>One of the above is required.
+In the case of the list id, data is returned for the last month.
+For email ID, the thread is returned.</p>
+<p>Response example:</p>
+<pre>
+TBA
+</pre>
 <div style="display: inline-block; background: #BBB; margin: -10px; padding: 10px;">
     <h4><a id="disclaimer"></a>Disclaimer</h4>
 <div style="width: 65%; float: left;">
diff --git a/content/docs/design-notes.html b/content/docs/design-notes.html
new file mode 100644
index 0000000..3baad8e
--- /dev/null
+++ b/content/docs/design-notes.html
@@ -0,0 +1,118 @@
+<!DOCTYPE html><html><head><meta charset="utf-8"><title>Apache Pony Mail (Incubating)</title>
+<link rel="stylesheet" type="text/css" href="/css/default.css"/>
+<link rel="stylesheet" type="text/css" href="/css/fa/fa.css"/>
+<link rel="apple-touch-icon" sizes="57x57" href="/icons/apple-icon-57x57.png">
+<link rel="apple-touch-icon" sizes="60x60" href="/icons/apple-icon-60x60.png">
+<link rel="apple-touch-icon" sizes="72x72" href="/icons/apple-icon-72x72.png">
+<link rel="apple-touch-icon" sizes="76x76" href="/icons/apple-icon-76x76.png">
+<link rel="apple-touch-icon" sizes="114x114" href="/icons/apple-icon-114x114.png">
+<link rel="apple-touch-icon" sizes="120x120" href="/icons/apple-icon-120x120.png">
+<link rel="apple-touch-icon" sizes="144x144" href="/icons/apple-icon-144x144.png">
+<link rel="apple-touch-icon" sizes="152x152" href="/icons/apple-icon-152x152.png">
+<link rel="apple-touch-icon" sizes="180x180" href="/icons/apple-icon-180x180.png">
+<link rel="icon" type="image/png" sizes="192x192"  href="/android-icon-192x192.png">
+<link rel="icon" type="image/png" sizes="32x32" href="/icons/favicon-32x32.png">
+<link rel="icon" type="image/png" sizes="96x96" href="/icons/favicon-96x96.png">
+<link rel="icon" type="image/png" sizes="16x16" href="/icons/favicon-16x16.png">
+<link rel="manifest" href="/icons/manifest.json">
+<meta name="msapplication-TileColor" content="#ffffff">
+<meta name="msapplication-TileImage" content="/icons/ms-icon-144x144.png">
+<meta name="theme-color" content="#ffffff">
+</head><body>
+<div id="titlebar">
+    <a href="/contribute.html"><img align='left' style="width: 130px; height: 125px; position: relative; left: -6px; top: -6px; border: 0;" src="/images/devme.png" alt="Fork/Hack on Pony Mail"></a>
+    <a href='/'><img src="/images/ponymail.svg" style="width: 110px; margin-left: -10px; margin-right: 20px; height: auto;" align="left"/></a>
+    <h1><a id="title" href="/" style="color: #FFF !important;">Apache Pony Mail&trade; (Incubating)</a></h1>
+    <div id="menubar">
+        <ul>
+            <li><a href="/docs.html"><i class="fa fa-book"></i><span>Documentation</span></a></li>
+            <li><a href="/source.html"><i class="fa fa-git-square"></i><span>Source</span></a></li>
+            <li><a href="/downloads.html"><i class="fa fa-cloud-download"></i><span>Download</span></a></li>
+            <li><a href="/support.html"><i class="fa fa-question-circle"></i><span>Get support</span></a></li>
+            <li><a href="/contribute.html"><i class="fa fa-share-alt"></i><span>Contribute</span></a></li>
+            <li><a href="/about.html"><i class="fa fa-users"></i><span>About</span></a></li>
+        </ul>
+    </div>
+</div>
+<h1 id='designnotes'>Design Notes<a href='#designnotes' style='color: rgba(0,0,0,0);'>&para;</a></h1>
+<p>This file is an attempt to summarise some of the design issues.</p>
+<h2 id='database'>Database<a href='#database' style='color: rgba(0,0,0,0);'>&para;</a></h2>
+<p>The project uses the ElasticSearch (ES) database to store the mails as individual documents.
+The database stores each mail to each list as a separate document.
+If the same mail was sent to multiple lists, then it exists as multiple documents in the database.</p>
+<p>ES requires that each distinct document has a unique id (MID).
+The MID is used to insert the document in the database, and can be used to fetch it.</p>
+<h3 id='databasedesign'>Database design<a href='#databasedesign' style='color: rgba(0,0,0,0);'>&para;</a></h3>
+<p>The mails are stored in two separate ES indexes:
+<em> "mbox" - this stores information about the document, plus the parsed content, and is used for searching and summary displays.
+</em> "mbox_source" - this is used to store the raw content of the document.
+The two versions of the document are linked by using the same MID.</p>
+<h3 id='requirementsforthemid'>Requirements for the MID<a href='#requirementsforthemid' style='color: rgba(0,0,0,0);'>&para;</a></h3>
+<p>As mentioned above, each different document must have a unique id (MID).
+This document may arrive as a single mail message, or be loaded from a collection such as an mbox file.</p>
+<p>Duplicate database entries can be avoided by ensuring that the same MID is calculated regardless of the input source.
+[If the same message is processed more than once, it then does not matter as only the last instance will be stored.]
+The MID format does not have to be transparent; it can be an opaque hash.</p>
+<h3 id='generationofthemid'>Generation of the MID<a href='#generationofthemid' style='color: rgba(0,0,0,0);'>&para;</a></h3>
+<p>The same message may be sent to multiple lists, so the message data alone is not sufficient to identify it uniquely.
+The same message may potentially be sent more than once to the same list,
+so the combination of message and listname is also not sufficient to identify a message.</p>
+<p>Many messages will have a Message-Id header which is intended to be unique to the message.
+However this may not be the case, and some messages do not have one.</p>
+<p>Many mailing list servers will allocate a squence number or other such id to each message they send.
+This should be unique for the list, assuming that sequence is not reset.</p>
+<p>Where the Message-Id and List Server Id both exist, they can be combined to generate a MID.
+[If the List Server Id is known to be unique, then that can potentially be used alone.] </p>
+<p>Where one or other id does not exist, then alternative means need to be used to generate the MID.
+The data used to do so must be present it all supported message sources.</p>
+<p>Algorithms for the generator remain TBA</p>
+<h3 id='permalinkrequirements'>Permalink requirements<a href='#permalinkrequirements' style='color: rgba(0,0,0,0);'>&para;</a></h3>
+<p>The application provides Permalinks which can later be used to refer to any document in the database.
+Once published, it is important that such links must continue to work.</p>
+<p>Links should be portable; i.e. if the raw messages are loaded into a new archive it should be possible
+to support existing published Permalinks.</p>
+<p>Multiple links may refer to the same document, however each link should refer to a single document.
+Ideally the Permalink should be relatively short; however that may conflict with the uniqueness requirement.</p>
+<p>It may be useful for the Permalink format to be relatively transparent.
+For example, a current ASF mod_mbox link looks like:</p>
+<p>http://mail-archives.apache.org/mod_mbox/ponymail-commits/201605.mbox/<a href="&#109;&#97;&#105;&#108;&#116;&#111;&#58;&#49;&#102;&#55;&#51;&#98;&#52;&#101;&#48;&#102;&#99;&#49;&#97;&#52;&#102;&#98;&#98;&#98;&#102;&#101;&#52;&#100;&#49;&#53;&#53;&#50;&#57;&#51;&#99;&#50;&#102;&#49;&#97;&#64;&#103;&#105;&#116;&#46;&#97;&#112;&#97;&#99;&#104;&#101;&#46;&#111;&#114;&#103;">&#49;&#102;&#55;&#51;&#98;&#52;&#101;&#48;&#102;&#99;&#49;&#97;&#52;&#102;&#98;&#98;&#98;&#102;&#101;&#52;&#100;&#49;&#53;&#53;&#50;&#57;&#51;&#99;&#50;&#102;&#49;&#97;&#64;&#103;&#105;&#116;&#46;&#97;&#112;&#97;&#99;&#104;&#101;&#46;&#111;&#114;&#103;</a></p>
+<p>This includes a reference to the:
+- mailing list name (ponymail-commits)
+- month when mail was sent (201605.mbox)
+- the Message-Id (<a href="&#109;&#97;&#105;&#108;&#116;&#111;&#58;&#49;&#102;&#55;&#51;&#98;&#52;&#101;&#48;&#102;&#99;&#49;&#97;&#52;&#102;&#98;&#98;&#98;&#102;&#101;&#52;&#100;&#49;&#53;&#53;&#50;&#57;&#51;&#99;&#50;&#102;&#49;&#97;&#64;&#103;&#105;&#116;&#46;&#97;&#112;&#97;&#99;&#104;&#101;&#46;&#111;&#114;&#103;">&#49;&#102;&#55;&#51;&#98;&#52;&#101;&#48;&#102;&#99;&#49;&#97;&#52;&#102;&#98;&#98;&#98;&#102;&#101;&#52;&#100;&#49;&#53;&#53;&#50;&#57;&#51;&#99;&#50;&#102;&#49;&#97;&#64;&#103;&#105;&#116;&#46;&#97;&#112;&#97;&#99;&#104;&#101;&#46;&#111;&#114;&#103;</a>)</p>
+<p>This information should be sufficient to find the message in just about any mail-archive.</p>
+<p>Whereas vendor-specific links may be much shorter, but are only valid for the particular service.
+For example the equivalent Markmail link is:
+http://markmail.org/message/oanktcpxlxkmyora</p>
+<p>There may be use cases for both styles of link.</p>
+<h3 id='permalinkdesign'>Permalink design<a href='#permalinkdesign' style='color: rgba(0,0,0,0);'>&para;</a></h3>
+<p>TBA</p>
+<div style="display: inline-block; background: #BBB; margin: -10px; padding: 10px;">
+    <h4><a id="disclaimer"></a>Disclaimer</h4>
+<div style="width: 65%; float: left;">
+<p style="line-height: 12pt;">
+    Apache Pony Mail (Incubating) is an effort undergoing incubation at
+    The Apache Software Foundation (ASF), sponsored by the <a href="https://incubator.apache.org">
+    Apache Incubator</a>. Incubation is required of all newly accepted projects
+    until a further review indicates that the infrastructure,
+    communications, and decision making process have stabilized in a
+    manner consistent with other successful ASF projects. While
+    incubation status is not necessarily a reflection of the
+    completeness or stability of the code, it does indicate that the
+    project has yet to be fully endorsed by the ASF.
+</p>
+</div>
+<div style="width: 20%; float: left; padding-top: 16px; margin-left: -8px;">
+<a href="https://incubator.apache.org"><img src="https://incubator.apache.org/images/incubator_feather_egg_logo_sm.png"/></a>
+</div>
+<div style="width: 15%; float: left; line-height: 15px; padding-left: 22px;">
+    <a class="item" target="_blank" href="https://incubator.apache.org/">Apache Incubator</a><br/>
+            <a class="item" target="_blank" href="https://www.apache.org/">About the ASF<span></span></a><br/>
+            <a class="item" target="_blank" href="https://www.apache.org/foundation/policies/conduct.html">Code of Conduct<span></span></a><br/>
+            <a class="item" target="_blank" href="https://www.apache.org/foundation/thanks.html">Thanks<span></span></a><br/>
+            <a class="item" target="_blank" href="https://www.apache.org/foundation/sponsorship.html">Become a Sponsor<span></span></a><br/>
+            <a class="item" target="_blank" href="https://www.apache.org/security/">Security<span></span></a><br/>
+            <a class="item" target="_blank" href="https://www.apache.org/licenses/LICENSE-2.0">License<span></span></a><br/>
+</div>
+</div>
+</body></html>
diff --git a/content/docs/installing.html b/content/docs/installing.html
index da5b7cf..4e63cb7 100644
--- a/content/docs/installing.html
+++ b/content/docs/installing.html
@@ -47,8 +47,9 @@
 <h2 id='prerequisites'>Pre-requisites<a href='#prerequisites' style='color: rgba(0,0,0,0);'>&para;</a></h2>
 <p>You will need the following software installed on your machine:</p>
 <ul>
-<li>ElasticSearch &gt;= 2.1</li>
+<li>ElasticSearch &gt;= 2.1 and &lt; 6.0 (setup.py does not support 6.x+; the code may perhaps run on 6.x)</li>
 <li>Python 3.x for the archiver plugin (setup.py will handle dependencies) and importer</li>
+<li>Python <code>html2text</code> package (GPLv3) if you wish to archive HTML-only mails (remember to add the <code>--html2text</code> command line arg)</li>
 <li>Apache HTTP Server 2.4.x with mod_lua (see http://modlua.org/gs/installing if you need to build mod_lua manually)</li>
 <li>Lua &gt;=5.1 with the following modules: cjson, luasec, luasocket
   (Note: Lua 5.3 is not currently supported by httpd mod_lua or luasocket)</li>
@@ -205,7 +206,14 @@
 <p>By default, headers such as to/cc are not shown in the normal email view.
 To enable these headers, set <code>full_headers</code> to <code>true</code> in the <code>site/api/lib/config.lua</code> file.</p>
 <h3 id='lastlyanoteaboutmessageidmidgenerators'>Lastly, a note about Message-ID (MID) generators<a href='#lastlyanoteaboutmessageidmidgenerators' style='color: rgba(0,0,0,0);'>&para;</a></h3>
-<p>Please see <a href="archiving.html#usingtherightidgenerator">this paragraph</a> about document ID generators.</p>
+<p>The default MID generator is called 'medium' and digests the message
+body, timestamp and list-ID to generate the MID. There is also a 'short'
+that only digests the body, and a 'full' that uses the entire message as
+a bytestring to generate an ID. Medium is recommended for most setups
+(especially clustered setups), while full can be used for single-machine
+setups.
+N.B. At present, all the generators have issues, see (#176 #177 #178)
+Please see <a href="archiving.html#usingtherightidgenerator">this paragraph</a> about document ID generators.</p>
 <div style="display: inline-block; background: #BBB; margin: -10px; padding: 10px;">
     <h4><a id="disclaimer"></a>Disclaimer</h4>
 <div style="width: 65%; float: left;">