Copy updates from main ponymail repo
diff --git a/source/markdown/docs/API.md b/source/markdown/docs/API.md
index a73213e..1d62f75 100644
--- a/source/markdown/docs/API.md
+++ b/source/markdown/docs/API.md
@@ -33,12 +33,15 @@
"tid": "06b318af97ca96c115e878c14d0814a53407751c31388410421c1751@1441467256@<dev.any23.apache.org>",
"list_raw": "<dev.any23.apache.org>"
}
+
+Note: date and epoch are in UTC
+
~~~
### Fetching list data
Usage:
-`GET /api/stats.lua?list=$list&domain=$domain[&d=$timespan][&q=$query][&header_from=$from][&header_to=$to][&header_subject=$subject][&header_body=$body][&quick][&emailsOnly][&s=$s&e=$e]`
+`GET /api/stats.lua?list=$list&domain=$domain[&d=$timespan][&q=$query][&header_from=$from][&header_to=$to][&header_subject=$subject][&header_body=$body][&quick][&emailsOnly][&s=$s&e=$e][&since=$since][&dfrom=$dfrom&dto=$dto]`
See below for details of [timespan](#Timespans) values
@@ -46,7 +49,7 @@
- $list: The list prefix (e.g. `dev`). Wildcards may be used
- $domain: The list domain (e.g. `httpd.apache.org`). Wildcards may be used
- - $timespan: A timespan value (see below)
+ - $timespan: A [timespan](#Timespans) value
- $s: yyyy-mm start of month (day 1)
- $e: yyyy-mm end of month (last day)
- $query: A search query (may contain wildcards or negations):
@@ -57,6 +60,13 @@
- $to: Optional To: address
- $subject: Optional Subject: line
- $body: Optional body text
+ - $since: number of seconds since the epoch, defaults to now.
+ Returns '{"changed":false}' if no emails are later than epoch, otherwise proceeds with normal search
+ - $dfrom: days ago to start
+ - $dto: total days to match
+
+Options:
+
- quick: send statistics only (exclude participants, threadstruct, word-cloud, emails apart from epoch)
- emailsOnly: return email summaries only (omit thread_struct, top 10 participants and word-cloud)
@@ -94,7 +104,35 @@
"name": "dev",
"cloud": {...},
"hits": 25,
- "thread_struct": {...},
+ thread_struct":
+ {
+ "nest": 2,
+ "children": {
+ {
+ "children": {
+ {
+ "children": {
+ {
+ "children": { },
+ epoch: ...,
+ tid: ...,
+ nest: 1
+ }
+ },
+ epoch: ...,
+ tid: ...,
+ nest: 2
+ }
+ },
+ "epoch": 1474883100,
+ "tid": "b1d6446f5cc8f4846454cbabc48ddb08afbb601a77169f8e32e34102@<dev.ponymail.apache.org>",
+ "nest": 2
+ }
+ },
+ epoch: ...,
+ tid: ...,
+ body: ...
+ },
"max": 5000,
"searchlist": "<dev.ponymail.info>",
"list": "dev@ponymail.info",
@@ -167,9 +205,10 @@
### Fetching notifications for a logged in user
Usage:
-`GET /api/notifications.lua`
+`GET /api/notifications.lua[?seen=$mid]`
-Parameters: `None` (cookie required)
+Parameters: (cookie required)
+ - $mid: id of the message to be marked as having been seen
Response example:
@@ -178,6 +217,8 @@
{
"notifications": {...}
}
+or
+{"marked": true}
~~~
### Fetching a month's data as an mbox file
@@ -190,3 +231,21 @@
TBA
~~~
+### Get ATOM data for list or email
+
+Usage:
+`GET /api/atom/lua(?list=$lid|?mid=$mid)`
+
+Parameters: (cookie may be required)
+ - $lid: the list id, e.g. dev@ponymail.apache.org
+ - $mid: The email ID (Permalink)
+
+One of the above is required.
+In the case of the list id, data is returned for the last month.
+For email ID, the thread is returned.
+
+Response example:
+
+~~~
+TBA
+~~~
diff --git a/source/markdown/docs/DESIGN-NOTES.md b/source/markdown/docs/DESIGN-NOTES.md
new file mode 100644
index 0000000..1feee34
--- /dev/null
+++ b/source/markdown/docs/DESIGN-NOTES.md
@@ -0,0 +1,76 @@
+# Design Notes
+
+This file is an attempt to summarise some of the design issues.
+
+## Database
+The project uses the ElasticSearch (ES) database to store the mails as individual documents.
+The database stores each mail to each list as a separate document.
+If the same mail was sent to multiple lists, then it exists as multiple documents in the database.
+
+ES requires that each distinct document has a unique id (MID).
+The MID is used to insert the document in the database, and can be used to fetch it.
+
+### Database design
+The mails are stored in two separate ES indexes:
+* "mbox" - this stores information about the document, plus the parsed content, and is used for searching and summary displays.
+* "mbox_source" - this is used to store the raw content of the document.
+The two versions of the document are linked by using the same MID.
+
+### Requirements for the MID
+As mentioned above, each different document must have a unique id (MID).
+This document may arrive as a single mail message, or be loaded from a collection such as an mbox file.
+
+Duplicate database entries can be avoided by ensuring that the same MID is calculated regardless of the input source.
+[If the same message is processed more than once, it then does not matter as only the last instance will be stored.]
+The MID format does not have to be transparent; it can be an opaque hash.
+
+### Generation of the MID
+The same message may be sent to multiple lists, so the message data alone is not sufficient to identify it uniquely.
+The same message may potentially be sent more than once to the same list,
+so the combination of message and listname is also not sufficient to identify a message.
+
+Many messages will have a Message-Id header which is intended to be unique to the message.
+However this may not be the case, and some messages do not have one.
+
+Many mailing list servers will allocate a squence number or other such id to each message they send.
+This should be unique for the list, assuming that sequence is not reset.
+
+Where the Message-Id and List Server Id both exist, they can be combined to generate a MID.
+[If the List Server Id is known to be unique, then that can potentially be used alone.]
+
+Where one or other id does not exist, then alternative means need to be used to generate the MID.
+The data used to do so must be present it all supported message sources.
+
+Algorithms for the generator remain TBA
+
+### Permalink requirements
+The application provides Permalinks which can later be used to refer to any document in the database.
+Once published, it is important that such links must continue to work.
+
+Links should be portable; i.e. if the raw messages are loaded into a new archive it should be possible
+to support existing published Permalinks.
+
+Multiple links may refer to the same document, however each link should refer to a single document.
+Ideally the Permalink should be relatively short; however that may conflict with the uniqueness requirement.
+
+It may be useful for the Permalink format to be relatively transparent.
+For example, a current ASF mod_mbox link looks like:
+
+http://mail-archives.apache.org/mod_mbox/ponymail-commits/201605.mbox/<1f73b4e0fc1a4fbbbfe4d155293c2f1a@git.apache.org>
+
+This includes a reference to the:
+- mailing list name (ponymail-commits)
+- month when mail was sent (201605.mbox)
+- the Message-Id (<1f73b4e0fc1a4fbbbfe4d155293c2f1a@git.apache.org>)
+
+This information should be sufficient to find the message in just about any mail-archive.
+
+Whereas vendor-specific links may be much shorter, but are only valid for the particular service.
+For example the equivalent Markmail link is:
+http://markmail.org/message/oanktcpxlxkmyora
+
+There may be use cases for both styles of link.
+
+### Permalink design
+TBA
+
diff --git a/source/markdown/docs/INSTALLING.md b/source/markdown/docs/INSTALLING.md
index de197ef..0ecd8d3 100644
--- a/source/markdown/docs/INSTALLING.md
+++ b/source/markdown/docs/INSTALLING.md
@@ -14,8 +14,9 @@
## Pre-requisites ##
You will need the following software installed on your machine:
-- ElasticSearch >= 2.1
+- ElasticSearch >= 2.1 and < 6.0 (setup.py does not support 6.x+; the code may perhaps run on 6.x)
- Python 3.x for the archiver plugin (setup.py will handle dependencies) and importer
+- Python `html2text` package (GPLv3) if you wish to archive HTML-only mails (remember to add the `--html2text` command line arg)
- Apache HTTP Server 2.4.x with mod_lua (see http://modlua.org/gs/installing if you need to build mod_lua manually)
- Lua >=5.1 with the following modules: cjson, luasec, luasocket
(Note: Lua 5.3 is not currently supported by httpd mod_lua or luasocket)
@@ -208,4 +209,11 @@
To enable these headers, set `full_headers` to `true` in the `site/api/lib/config.lua` file.
### Lastly, a note about Message-ID (MID) generators
+The default MID generator is called 'medium' and digests the message
+body, timestamp and list-ID to generate the MID. There is also a 'short'
+that only digests the body, and a 'full' that uses the entire message as
+a bytestring to generate an ID. Medium is recommended for most setups
+(especially clustered setups), while full can be used for single-machine
+setups.
+N.B. At present, all the generators have issues, see (#176 #177 #178)
Please see [this paragraph](archiving.html#usingtherightidgenerator) about document ID generators.
