Merge branch 'apexcore-514.apexwebsite' of https://github.com/mx4web/apex-site
diff --git a/releases.json b/releases.json
index c24bfdd..b28219d 100644
--- a/releases.json
+++ b/releases.json
@@ -12,18 +12,18 @@
"docs": "3.2",
"repo": "apex-core",
"date": 1461193940000
- },
- {
- "version": "3.3.0-incubating",
- "docs": "3.3",
- "repo": "apex-core",
- "date": 1454908501000
}
]
},
"malhar": {
"src": [
{
+ "version": "3.5.0",
+ "docs": "3.5",
+ "repo": "apex-malhar",
+ "date": 1472622216000
+ },
+ {
"version": "3.4.0",
"docs": "3.3",
"repo": "apex-malhar",
diff --git a/roadmap.json b/roadmap.json
index f517307..3a6bcb2 100644
--- a/roadmap.json
+++ b/roadmap.json
@@ -284,12 +284,12 @@
},
{
"expand": "operations,editmeta,changelog,transitions,renderedFields",
- "id": "12955374",
- "self": "https://issues.apache.org/jira/rest/api/2/issue/12955374",
- "key": "APEXCORE-414",
+ "id": "12955475",
+ "self": "https://issues.apache.org/jira/rest/api/2/issue/12955475",
+ "key": "APEXCORE-418",
"fields": {
- "summary": "Native support for event-time windowing",
- "description": "Apex core has streaming windows that establish a boundary based on arrival time of events. Many applications require boundaries based on the time of events, which could be a field in the tuple. Some of the operators support this today (time bucketing), but it would be good to provide more generic support for this in the engine itself. ",
+ "summary": "Support for Mesos",
+ "description": "Today Apex has two modes of execution: Embedded mode (everything running in a single JVM) and YARN. There has been a few questions around native support for Mesos. A cursory look suggests that Mesos support can be added by reimplementing the YARN specific portions in the master (AppMasterService, ContainerLauncher) and limited changes to the streaming container driver.\r\n\r\nMesos has a different model of resource allocation: The master offers resources to the framework while in YARN resources are requested. Apex master needs to implement the \"framework scheduler\" that is responsible to accept the resources and control the tasks.\r\n\r\nhttp://mesos.apache.org/documentation/latest/app-framework-development-guide/\r\n\r\nTasks are launched through executors, command line and docker executors are provided. \r\n\r\nApex also requires support to deploy the dependencies to the nodes on which the streaming containers are launched. YARN supports that through resource localization. Mesos supports this through the fetcher, which can copy the resources to the slave node.\r\n\r\nhttp://mesos.apache.org/documentation/latest/fetcher/\r\n",
"fixVersions": [],
"priority": {
"self": "https://issues.apache.org/jira/rest/api/2/priority/3",
@@ -315,12 +315,43 @@
},
{
"expand": "operations,editmeta,changelog,transitions,renderedFields",
- "id": "12955475",
- "self": "https://issues.apache.org/jira/rest/api/2/issue/12955475",
- "key": "APEXCORE-418",
+ "id": "12992322",
+ "self": "https://issues.apache.org/jira/rest/api/2/issue/12992322",
+ "key": "APEXCORE-498",
"fields": {
- "summary": "Support for Mesos",
- "description": "Today Apex has two modes of execution: Embedded mode (everything running in a single JVM) and YARN. There has been a few questions around native support for Mesos. A cursory look suggests that Mesos support can be added by reimplementing the YARN specific portions in the master (AppMasterService, ContainerLauncher) and limited changes to the streaming container driver.\r\n\r\nMesos has a different model of resource allocation: The master offers resources to the framework while in YARN resources are requested. Apex master needs to implement the \"framework scheduler\" that is responsible to accept the resources and control the tasks.\r\n\r\nhttp://mesos.apache.org/documentation/latest/app-framework-development-guide/\r\n\r\nTasks are launched through executors, command line and docker executors are provided. \r\n\r\nApex also requires support to deploy the dependencies to the nodes on which the streaming containers are launched. YARN supports that through resource localization. Mesos supports this through the fetcher, which can copy the resources to the slave node.\r\n\r\nhttp://mesos.apache.org/documentation/latest/fetcher/\r\n",
+ "summary": "Named Checkpoints - Checkpoint the DAG with a name/tag and start the app from that point",
+ "description": "Named Checkpoints \r\n\r\n1. Ability to tag/name the checkpoints\r\n2. On demand - checkpoint the DAG\r\n3. Start the app from the named checkpoints\r\n\r\nAll checkpoints that happened before the committed window is deleted but the named checkpoints won't be deleted.",
+ "fixVersions": [],
+ "priority": {
+ "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
+ "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
+ "name": "Major",
+ "id": "3"
+ },
+ "status": {
+ "self": "https://issues.apache.org/jira/rest/api/2/status/1",
+ "description": "The issue is open and ready for the assignee to start work on it.",
+ "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/open.png",
+ "name": "Open",
+ "id": "1",
+ "statusCategory": {
+ "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/2",
+ "id": 2,
+ "key": "new",
+ "colorName": "blue-gray",
+ "name": "New"
+ }
+ }
+ }
+ },
+ {
+ "expand": "operations,editmeta,changelog,transitions,renderedFields",
+ "id": "13005178",
+ "self": "https://issues.apache.org/jira/rest/api/2/issue/13005178",
+ "key": "APEXCORE-536",
+ "fields": {
+ "summary": "Upgrade Hadoop dependency",
+ "description": "Currently Apex depends on Hadoop 2.2 and runs on all later 2.x version. Hadoop 2.2 is quite old, most Apex users have more recent Hadoop installs. Latest distro releases are based on 2.6 and 2.7. There are several important features that were added in Hadoop since 2.2 that Apex should be able to leverage.",
"fixVersions": [],
"priority": {
"self": "https://issues.apache.org/jira/rest/api/2/priority/3",
@@ -402,37 +433,6 @@
"jiras": [
{
"expand": "operations,editmeta,changelog,transitions,renderedFields",
- "id": "12926249",
- "self": "https://issues.apache.org/jira/rest/api/2/issue/12926249",
- "key": "APEXMALHAR-1720",
- "fields": {
- "summary": "Development of Inner Join Operator",
- "description": null,
- "fixVersions": [],
- "priority": {
- "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
- "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
- "name": "Major",
- "id": "3"
- },
- "status": {
- "self": "https://issues.apache.org/jira/rest/api/2/status/3",
- "description": "This issue is being actively worked on at the moment by the assignee.",
- "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/inprogress.png",
- "name": "In Progress",
- "id": "3",
- "statusCategory": {
- "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/4",
- "id": 4,
- "key": "indeterminate",
- "colorName": "yellow",
- "name": "In Progress"
- }
- }
- }
- },
- {
- "expand": "operations,editmeta,changelog,transitions,renderedFields",
"id": "12926159",
"self": "https://issues.apache.org/jira/rest/api/2/issue/12926159",
"key": "APEXMALHAR-1811",
@@ -478,17 +478,17 @@
"id": "3"
},
"status": {
- "self": "https://issues.apache.org/jira/rest/api/2/status/1",
- "description": "The issue is open and ready for the assignee to start work on it.",
- "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/open.png",
- "name": "Open",
- "id": "1",
+ "self": "https://issues.apache.org/jira/rest/api/2/status/3",
+ "description": "This issue is being actively worked on at the moment by the assignee.",
+ "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/inprogress.png",
+ "name": "In Progress",
+ "id": "3",
"statusCategory": {
- "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/2",
- "id": 2,
- "key": "new",
- "colorName": "blue-gray",
- "name": "New"
+ "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/4",
+ "id": 4,
+ "key": "indeterminate",
+ "colorName": "yellow",
+ "name": "In Progress"
}
}
}
@@ -588,45 +588,6 @@
},
{
"expand": "operations,editmeta,changelog,transitions,renderedFields",
- "id": "12953482",
- "self": "https://issues.apache.org/jira/rest/api/2/issue/12953482",
- "key": "APEXMALHAR-2026",
- "fields": {
- "summary": "Spill-able Datastructures",
- "description": "Add libraryies for spooling datastructures to a key value store. There are several customer use cases which require spooled data structures.\r\n\r\n1 - Some operators like AbstractFileInputOperator have ever growing state. This is an issue because eventually the state of the operator will grow larger than the memory allocated to the operator, which will cause the operator to perpetually fail. However if the operator's datastructures are spooled then the operator will never run out of memory.\r\n\r\n2 - Some users have requested for the ability to maintain a map as well as a list of keys over which to iterate. Most key value stores don't provide this functionality. However, with spooled datastructures this functionality can be provided by maintaining a spooled map and an iterable set of keys.\r\n\r\n3 - Some users have requested building graph databases within APEX. This would require implementing a spooled graph data structure.\r\n\r\n4 - Another use case for spooled data structures is database operators. Database operators need to write data to a data base, but sometimes the database is down. In this case most of the database operators repeatedly fail until the database comes back up. In order to avoid constant failures the database operator need to writes data to a queue when the data base is down, then when the database is up the operator need to take data from the queue and write it to the database. In the case of a database failure this queue will grow larger than the total amount of memory available to the operator, so the queue should be spooled in order to prevent the operator from failing.\r\n\r\n5 - Any operator which needs to maintain a large data structure in memory currently needs to have that data serialized and written out to HDFS with every checkpoint. This is costly when the data structure is large. If the data structure is spooled, then only the changes to the data structure are written out to HDFS instead of the entire data structure.\r\n\r\n6 - Also building an Apex Native database for aggregations requires indices. These indices need to take the form of spooled data structures.\r\n\r\n7 - In the future any operator which needs to maintain a data structure larger than the memory available to it will need to spool the data structure.",
- "fixVersions": [
- {
- "self": "https://issues.apache.org/jira/rest/api/2/version/12335815",
- "id": "12335815",
- "name": "3.5.0",
- "archived": false,
- "released": false
- }
- ],
- "priority": {
- "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
- "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
- "name": "Major",
- "id": "3"
- },
- "status": {
- "self": "https://issues.apache.org/jira/rest/api/2/status/5",
- "description": "A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.",
- "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/resolved.png",
- "name": "Resolved",
- "id": "5",
- "statusCategory": {
- "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/3",
- "id": 3,
- "key": "done",
- "colorName": "green",
- "name": "Complete"
- }
- }
- }
- },
- {
- "expand": "operations,editmeta,changelog,transitions,renderedFields",
"id": "12969033",
"self": "https://issues.apache.org/jira/rest/api/2/issue/12969033",
"key": "APEXMALHAR-2089",
@@ -658,17 +619,17 @@
},
{
"expand": "operations,editmeta,changelog,transitions,renderedFields",
- "id": "12988875",
- "self": "https://issues.apache.org/jira/rest/api/2/issue/12988875",
- "key": "APEXMALHAR-2142",
+ "id": "12985430",
+ "self": "https://issues.apache.org/jira/rest/api/2/issue/12985430",
+ "key": "APEXMALHAR-2130",
"fields": {
- "summary": "High-level API window support",
- "description": null,
+ "summary": "Scalable windowed storage",
+ "description": "This feature is used for supporting windowing.\r\n\r\nThe storage needs to have the following features:\r\n1. Spillable key value storage (integrate with APEXMALHAR-2026)\r\n2. Upon checkpoint, it saves a snapshot for the entire data set with the checkpointing window id. This should be done incrementally (ManagedState) to avoid wasting space with unchanged data\r\n3. When recovering, it takes the recovery window id and restores to that snapshot\r\n4. When a window is committed, all windows with a lower ID should be purged from the store.\r\n5. It should implement the WindowedStorage and WindowedKeyedStorage interfaces, and because of 2 and 3, we may want to add methods to the WindowedStorage interface so that the implementation of WindowedOperator can notify the storage of checkpointing, recovering and committing of a window.\r\n",
"fixVersions": [
{
- "self": "https://issues.apache.org/jira/rest/api/2/version/12335815",
- "id": "12335815",
- "name": "3.5.0",
+ "self": "https://issues.apache.org/jira/rest/api/2/version/12338174",
+ "id": "12338174",
+ "name": "3.6.0",
"archived": false,
"released": false
}
@@ -680,6 +641,68 @@
"id": "3"
},
"status": {
+ "self": "https://issues.apache.org/jira/rest/api/2/status/4",
+ "description": "This issue was once resolved, but the resolution was deemed incorrect. From here issues are either marked assigned or resolved.",
+ "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/reopened.png",
+ "name": "Reopened",
+ "id": "4",
+ "statusCategory": {
+ "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/2",
+ "id": 2,
+ "key": "new",
+ "colorName": "blue-gray",
+ "name": "New"
+ }
+ }
+ }
+ },
+ {
+ "expand": "operations,editmeta,changelog,transitions,renderedFields",
+ "id": "13006875",
+ "self": "https://issues.apache.org/jira/rest/api/2/issue/13006875",
+ "key": "APEXMALHAR-2260",
+ "fields": {
+ "summary": "Python execution for operator logic ",
+ "description": "Support execution of Python code in an operator. \r\n\r\nhttps://lists.apache.org/thread.html/9837b1dee8f909ed400c6030ce5c6a94a12f43183718019dd0bfd228@%3Cdev.apex.apache.org%3E\r\n",
+ "fixVersions": [],
+ "priority": {
+ "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
+ "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
+ "name": "Major",
+ "id": "3"
+ },
+ "status": {
+ "self": "https://issues.apache.org/jira/rest/api/2/status/1",
+ "description": "The issue is open and ready for the assignee to start work on it.",
+ "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/open.png",
+ "name": "Open",
+ "id": "1",
+ "statusCategory": {
+ "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/2",
+ "id": 2,
+ "key": "new",
+ "colorName": "blue-gray",
+ "name": "New"
+ }
+ }
+ }
+ },
+ {
+ "expand": "operations,editmeta,changelog,transitions,renderedFields",
+ "id": "13006876",
+ "self": "https://issues.apache.org/jira/rest/api/2/issue/13006876",
+ "key": "APEXMALHAR-2261",
+ "fields": {
+ "summary": "Python binding for high level API",
+ "description": "A high level API similar to the Apex Java stream API that lets users specify an application in Python.\r\n\r\nhttps://lists.apache.org/thread.html/9837b1dee8f909ed400c6030ce5c6a94a12f43183718019dd0bfd228@%3Cdev.apex.apache.org%3E\r\n",
+ "fixVersions": [],
+ "priority": {
+ "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
+ "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
+ "name": "Major",
+ "id": "3"
+ },
+ "status": {
"self": "https://issues.apache.org/jira/rest/api/2/status/1",
"description": "The issue is open and ready for the assignee to start work on it.",
"iconUrl": "https://issues.apache.org/jira/images/icons/statuses/open.png",
@@ -717,22 +740,22 @@
"self": "https://issues.apache.org/jira/rest/api/2/version/12334968",
"id": "12334968",
"name": "3.3.2",
- "archived": false,
+ "archived": true,
"released": false,
"projectId": 12318824
},
{
"self": "https://issues.apache.org/jira/rest/api/2/version/12335827",
"id": "12335827",
- "name": "3.4.1",
+ "name": "3.5.1",
"archived": false,
"released": false,
"projectId": 12318824
},
{
- "self": "https://issues.apache.org/jira/rest/api/2/version/12335815",
- "id": "12335815",
- "name": "3.5.0",
+ "self": "https://issues.apache.org/jira/rest/api/2/version/12338174",
+ "id": "12338174",
+ "name": "3.6.0",
"archived": false,
"released": false,
"projectId": 12318824
diff --git a/src/md/announcements-history.md b/src/md/announcements-history.md
index 8c4da9f..f6766c6 100644
--- a/src/md/announcements-history.md
+++ b/src/md/announcements-history.md
@@ -5,10 +5,3 @@
Changes: https://github.com/apache/apex-core/blob/v3.2.1-incubating/CHANGELOG.md
-
-## Apache Apex Malhar Library release 3.3.1-incubating
-
-Mar 2, 2016. The Apache Apex community is pleased to announce release [3.3.1-incubating](/downloads.html) of the Malhar Library.
-
-Changes: https://github.com/apache/apex-malhar/blob/v3.3.1-incubating/CHANGELOG.md
-
diff --git a/src/md/announcements-latest.md b/src/md/announcements-latest.md
index c8c22a6..dbb5bef 100644
--- a/src/md/announcements-latest.md
+++ b/src/md/announcements-latest.md
@@ -1,16 +1,22 @@
# Announcements
+## Apache Apex Malhar Library release 3.5.0
+
+Sep 05, 2016. The Apache Apex community is pleased to announce release [3.5.0](/downloads.html) of the Malhar Library.
+
+Changes: https://github.com/apache/apex-malhar/blob/v3.5.0/CHANGELOG.md
+
## Apache Apex Malhar Library release 3.4.0
May 25, 2016. The Apache Apex community is pleased to announce release [3.4.0](/downloads.html) of the Malhar Library.
-Changes: https://github.com/apache/apex-core/blob/v3.4.0/CHANGELOG.md
+Changes: https://github.com/apache/apex-malhar/blob/v3.4.0/CHANGELOG.md
## Apache Apex Core release 3.4.0
May 12, 2016. The Apache Apex community is pleased to announce release [3.4.0](/downloads.html) of the Core.
-Changes: https://github.com/apache/apex-malhar/blob/v3.4.0/CHANGELOG.md
+Changes: https://github.com/apache/apex-core/blob/v3.4.0/CHANGELOG.md
## Apache Apex announced as Top-Level Project
diff --git a/src/md/community.md b/src/md/community.md
index 92beeea..eea6767 100644
--- a/src/md/community.md
+++ b/src/md/community.md
@@ -25,7 +25,7 @@
# Contributing
-Looking for ideas to get involved? Please see [JIRA tickets for newcomers](https://issues.apache.org/jira/issues/?jql=project%20in%20%28APEXCORE%2C%20APEXMALHAR%29%20and%20labels%20%3D%20newbie) and pick a ticket. Please also sign up to the dev mailing list and JIRA.
+Looking for ideas to get involved? Please see [JIRA tickets for newcomers](https://issues.apache.org/jira/issues/?jql=project%20in%20%28APEXCORE%2C%20APEXMALHAR%29%20and%20resolution%20%3D%20Unresolved%20and%20labels%20%3D%20newbie) and pick a ticket. Please also sign up to the dev mailing list and JIRA.
**To learn more about contributing to the project, [check out the contributing guidelines](/contributing.html).**
diff --git a/src/md/contributing.md b/src/md/contributing.md
index d542216..15d6e7d 100644
--- a/src/md/contributing.md
+++ b/src/md/contributing.md
@@ -14,37 +14,60 @@
People that help with the project in any of the above categories or other ways are contributors. See the [roles](http://www.apache.org/foundation/how-it-works.html#roles) as defined by the ASF. Community members that make sustained, welcome contributions to the project may be invited to become a [committer](/people.html).
-## Code Style
+## One-time Setup
-Apache Apex follows coding style that is closest to K & R style and uses [Checkstyle](http://checkstyle.sourceforge.net/) tool to enforce these standards. Travis CI will fail for any pull request that introduces any style violations.
+### JIRA
-The checkstyle configuration that Apache Apex projects use is present here : https://github.com/apache/apex-core/blob/master/codestyle-config/src/main/resources/apex_checks.xml
+Apache JIRA is used for issue tracking. If you do not already have an Apache JIRA account, sign up [here](https://issues.apache.org/jira/). Note that the user name should have no white spaces or other special characters that complicate auto-completion within JIRA comments etc.
+
+Please use a single JIRA account only (don't create multiple with different email addresses) to retain the issue history. Please use a permanent email address, for an existing account it can be changed in the profile. If you absolutely have to change your user name, contact INFRA.
+
+Apex has 2 JIRA projects:
+
+1. [APEXCORE](https://issues.apache.org/jira/browse/APEXCORE/) for [apex-core](https://github.com/apache/apex-core) and [apex-site](https://github.com/apache/apex-site)
+2. [APEXMALHAR](https://issues.apache.org/jira/browse/APEXMALHAR/) for [apex-malhar](https://github.com/apache/apex-malhar)
+
+Before working on changes for any of the repositories, please locate an existing JIRA ticket or submit a new one. In order to assign an issue to yourself, you need to be listed as contributor in the JIRA project. PMC members have access to add new contributors, please request to be added through a comment on your candidate ticket or send us an email on the dev@ mailing list.
+
+### Github and git
+
+We use GitHub’s pull request functionality to review proposed code changes. If you do not already have a personal GitHub account, sign up [here](https://github.com/join). We recommend that you use the same email address and first/lastname for emails, git and JIRA so that contributions can be better tracked and notifications correlated. It is also recommended that you use an email address that is valid permanently (for example your @gmail.com or @apache.org address). Please also see:
+
+* https://help.github.com/articles/setting-your-email-in-git/
+* https://help.github.com/articles/adding-an-email-address-to-your-github-account/
+* https://help.github.com/articles/keeping-your-email-address-private/
+
+If you are new to git, this [tutorial](https://try.github.io/) may be helpful.
+
+The ASF Apex git repositories have mirror repositories on github which are used to review pull requests and provide a second remote endpoint for the codebase.
+
+1. Fork the ASF github mirror: https://github.com/apache/apex-core (or https://github.com/apache/apex-malhar or https://github.com/apache/apex-site)
+1. Clone the **fork** on your local workspace (one time step):<br/>
+ `git clone https://github.com/{github_username}/apex-core.git`
+1. Add the mirror as remote repository, here shown for apex-core:<br/>
+ `git remote add upstream https://github.com/apache/apex-core`
+
+### Java Development Environment
+
+For prerequisites and setup instructions see: http://apex.apache.org/docs/apex/apex_development_setup/
+
+Apache Apex follows coding style that is closest to K & R style and uses [Checkstyle](http://checkstyle.sourceforge.net/) tool to enforce these standards. Apex checkstyle enforces no trailing whitespace. Travis CI will fail for any pull request that introduces any style violations. The checkstyle configuration is [here](https://github.com/apache/apex-core/blob/master/codestyle-config/src/main/resources/apex_checks.xml).
To make it easier for the users to set up their development environment, settings for the following common IDEs are provided in the Apache Apex Core repository with instructions.
- [IntelliJ](https://github.com/apache/apex-core/tree/master/misc/ide-templates/intellij)
- [Eclipse](https://github.com/apache/apex-core/tree/master/misc/ide-templates/eclipse)
- [NetBeans](https://github.com/apache/apex-core/tree/master/misc/ide-templates/netbeans)
-## Opening Pull Requests (contributors)
+## Opening Pull Requests
-The apex-core and apex-malhar repositories both have mirror repositories on github which are used to review pull requests and provide a second remote endpoint for the codebase.
-
-1. Create/assign a JIRA ([-core](https://issues.apache.org/jira/browse/APEXCORE/),[-malhar](https://issues.apache.org/jira/browse/APEXMALHAR/)) for the work you plan to do (or assign yourself to an existing JIRA ticket)
-1. Fork the ASF github mirror (one time step):
- https://github.com/apache/apex-core/
-1. Clone the **fork** on your local workspace (one time step):
- `git clone https://github.com/{github_username}/apex-core.git`
-1. Add [apex core](https://github.com/apache/apex-core) as a remote repository (one time step):
-`git remote add upstream https://github.com/apache/apex-core`
-1. Ensure that your git user name and email are [configured](https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup#Your-Identity), this will make it much easier to keep track of contributors (one time step).
-1. Create a new branch from the [master](https://github.com/apache/apex-core/tree/master) branch. **Name your branch with the JIRA number in it, e.g. `APEXCORE-123.my-feature`.**
-`git checkout -b APEXCORE-123.my-feature -t upstream/master`
-Creating a local branch that tracks a remote makes pull easier (no need to specify the remote branch while pulling). A branch can be made to track a remote branch anytime, not necessarily at its creation by:
+1. Create a new branch from the `master` branch, **name it with the JIRA number, e.g. `APEXCORE-123.my-feature`**:<br/>
+`git checkout -b APEXCORE-123.my-feature -t upstream/master`<br/>
+Creating a local branch that tracks a remote makes pull easier (no need to specify the remote branch while pulling). A branch can be made to track a remote branch anytime, not necessarily at its creation by:<br/>
`git branch -u upstream/master`
1. When adding new files, please include the Apache v2.0 license header.
- From the top level directory, run `mvn license:check -Dlicense.skip=false` to check correct header formatting.
- Run `mvn license:format -Dlicense.skip=false` to automatically add the header when missing.
-1. Once your feature is complete, submit the pull request on github against `master`.
+1. Once your feature is complete, submit the pull request on github against `master`. Use the JIRA number (e.g. APEXCORE-123) as prefix in the pull request title. This will ensure the information is attached to the JIRA ticket automatically.
1. If you want specific people to review your pull request, use the `@` notation in Github comments to mention that user, and request that he/she reviews your changes.
1. Check the status of the pull request and ensure the Travis CI build is successful. If not, inspect the linked build log for details.
- If build fails due to license headers, follow instructions above.
@@ -52,7 +75,7 @@
1. Add changes after the PR was opened to the same branch, Travis CI will detect changes and build automatically. To force the CI run, close and re-open the PR.
1. After all review is complete, combine all new commits into one squashed commit except when there are multiple contributors, and include the Jira number in the commit message. There are several ways to squash commits, but [here is one explanation from git-scm.com](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#Squashing-Commits) and a simple example is illustrated below:
- If tracking upstream/master then run `git rebase -i`. Else run `git rebase -i upstream/master`.
+ If tracking upstream/master then run `git rebase -i`. Else run `git rebase -i upstream/master`.
This command opens the text editor which lists the multiple commits:
```
@@ -105,7 +128,8 @@
- Travis CI pull request build needs to pass
- Ensure tests are added/modified for new features or fixes
- Ensure appropriate JavaDoc comments have been added
-1. To set up access to the ASF source repository, [follow these steps](https://git-wip-us.apache.org/#committers-getting-started). The ASF master repository is: `https://git-wip-us.apache.org/repos/asf/apex-core.git`
+1. To set up access to the ASF source repository, [follow these steps](https://git-wip-us.apache.org/#committers-getting-started). The ASF master repository is:<br/>
+ `https://git-wip-us.apache.org/repos/asf/apex-core.git`
1. Use the git command line to pull in the changes from the pull requests. You can refer to the corresponding email that will be automatically sent to the `dev@apex.apache.org` mailing list to see the exact commands to merge the given pull request.
1. Once done with verification, push the changes to the ASF repository's `master` branch. Within a few
seconds, the changes will propagate back to the github mirror and the pull requests be closed and marked merged automatically.
diff --git a/src/md/docs.md b/src/md/docs.md
index 02d8ee4..851db93 100644
--- a/src/md/docs.md
+++ b/src/md/docs.md
@@ -10,6 +10,7 @@
- <a href="http://docs.datatorrent.com/beginner/" rel="nofollow">Beginner's Guide to Apache Apex</a> This document provides a comprehensive overview of Apex and is recommended for developers just starting out with Apex.
- [Building Your First Apache Apex Application](https://youtu.be/LwRWBudOjg4) This video has a hands-on demonstration of how to check out the source code repositories and build them, then run the maven archetype command to generate a new Apache Apex project, populate the project with Java source files for a new application, and finally, build and run the application -- all on a virtual machine running Linux with Apache Hadoop installed.
+- [Writing an Apache Apex application](http://files.meetup.com/18978602/University%20program%20-%20Writing%20an%20Apache%20Apex%20application.pdf) A PDF document that frames a hands-on exercise of building a basic application; also includes a diagram illustrating the life-cycle of operators.
- <a href="http://docs.datatorrent.com/tutorials/topnwords/" rel="nofollow">Top N Words Application Tutorial</a> This document provides a detailed step-by-step description of how to build and run a
word counting application with Apache Apex starting with setting up your development environment, progressing to building, running and monitoring the application, visualizing the output and concluding with some advanced features such as assessing operator memory requirements, partitioning, and debugging.
- <a href="http://docs.datatorrent.com/tutorials/salesdimensions/" rel="nofollow">Sales Dimensions Application Tutorial</a> Similar to the Top N Words application but covers
@@ -20,12 +21,11 @@
### Presentations
- [Slideshare/ApacheApex](http://www.slideshare.net/ApacheApex/presentations) Presentations from past meetup events and other talks covering Apache Apex introduction, feature deep dive, integration, customer use cases and more.
-- [Writing an Apache Apex application](http://files.meetup.com/18978602/University%20program%20-%20Writing%20an%20Apache%20Apex%20application.pdf) A PDF document that frames a hands-on exercise of building a basic application; also includes a diagram illustrating the life-cycle of operators.
- [Next Gen Decision Making in < 2ms](https://www.youtube.com/watch?v=98EW5NGM3u0) A video discussing CapitalOne's experience with Apache Apex and evaluation of competing technologies along with the [slides](http://www.slideshare.net/ApacheApex/capital-ones-next-generation-decision-in-less-than-2-ms).
-- [Apache Nifi Integration with Apex](https://www.youtube.com/watch?v=EdBiOnQn3Gw) video and [slide deck](http://www.slideshare.net/ApacheApex/integrating-ni-fiandapex-by-bryan-bende).
- [Introducing Apache Apex](https://www.brighttalk.com/webcast/13685/190407) A webinar that begins with the historical context for the rise of Hadoop and Big Data, discusses why the promise of Hadoop remains largely unfulfilled and why moving beyond Map-Reduce model is essential and why operability is critically important. It continues with a discussion of the programming model, the various components of a running application on a YARN cluster and the large library of operators and connectors available with Apache Apex for reading data from and writing data to external systems. Concludes with a brief description of the visualization dashboards.
-- [Stream Processing with Apache Apex](http://www.slideshare.net/PramodImmaneni/meetup-59089806) A broad overview slide deck covering topics such as windowing, static and dynamic partitioning, unification, fault tolerance, locality, monitoring, etc.
-- [Fault Tolerance and Processing Semantics](https://www.brighttalk.com/webcast/13685/194115) A webinar and associated [slides](http://www.slideshare.net/ApacheApexOrganizer/webinar-fault-toleranceandprocessingsemantics) covering core Apache Apex features including checkpointing and fault tolerance with fast, incremental recovery via a buffer server which uses a publish-subscribe model for inter-operator data transport. A variety of failure scenarios and processing guarantees are discussed.
+- [Stream Processing with Apache Apex (video)](https://www.youtube.com/watch?v=1DVMSRTNdIQ) and [(slides)](http://www.slideshare.net/ApacheApex/hadoop-summit-sj-2016-next-gen-big-data-analytics-with-apache-apex) A broad overview slide deck covering topics such as windowing, static and dynamic partitioning, unification, fault tolerance, locality, monitoring, etc.
+- [Fault Tolerance and Processing Semantics (video)](https://www.youtube.com/watch?v=FCMY6Ii89Nw) and [(slides)](http://www.slideshare.net/ApacheApexOrganizer/webinar-fault-toleranceandprocessingsemantics) A webinar covering core Apache Apex features including checkpointing and fault tolerance with fast, incremental recovery via a buffer server which uses a publish-subscribe model for inter-operator data transport. A variety of failure scenarios and processing guarantees are discussed.
+- [Smart Partitioning with Apache Apex (video)](https://www.youtube.com/watch?v=kJWMajIjGG0) and [(slides)](http://www.slideshare.net/ApacheApex/smart-partitioning-with-apache-apex-webinar) Webinar covering partitioning, including unique Apex features such as elasticity with dynamic resource allocation, parallel partitions for speculative execution and processing SLA etc.
- [Windows in Apache Apex](http://www.slideshare.net/DevendraVyavahare/windowing-in-apex) Discusses the various flavors of windows available in Apache Apex and how to configure and
use them via callbacks. Contrasts windows with micro-batches.
- [Real Time Stream Processing Versus Batch](http://www.slideshare.net/DevendraVyavahare/batch-processing-vs-real-time-data-processing-streaming) Slide deck compares and contrasts the needs, use cases and challenges of stream processing with those of batch processing.
diff --git a/src/md/malhar-contributing.md b/src/md/malhar-contributing.md
index 218f167..3994d7f 100644
--- a/src/md/malhar-contributing.md
+++ b/src/md/malhar-contributing.md
@@ -25,11 +25,11 @@
## Implementing an operator
-* Look at the [Operator Development Guide](/docs/apex/operator_development) and the [Best Practices Guide](/docs/malhar/development_best_practices) on how to implement an operator and what the dos and don'ts are.
+* Look at the [Operator Development Guide](/docs/apex/operator_development) and the [Best Practices Guide](/docs/apex/development_best_practices) on how to implement an operator and what the dos and don'ts are.
* Refer to existing operator implementations when in doubt or unsure about how to implement some functionality. You can also email the [dev mailing list](/community.html#mailing-lists) with any questions.
* Write unit tests for operators
* Refer to unit tests for existing operators.
* If possible write a sample application testing the operator.
* Try to keep the tests self contained i.e., avoid the user having to perform additional setup or setup other services before running the tests. This can be done by starting mock versions of the required services from within the test and shutting them down when the test finishes.
* Follow the code style guidelines of the Apache Apex project described [here](http://apex.apache.org/contributing.html#code-style).
-* If only the core operator business logic for the operator is implemented and advanced platform functionality for scalability and recovery, such as partitioning, idempotence etc., described in the above reference guides are not implemented, make the code submission pull request to the **contrib** folder in Malhar, otherwise make it to an appropriate top-level module or a new one if one is not present.
\ No newline at end of file
+* If only the core operator business logic for the operator is implemented and advanced platform functionality for scalability and recovery, such as partitioning, idempotence etc., described in the above reference guides are not implemented, make the code submission pull request to the **contrib** folder in Malhar, otherwise make it to an appropriate top-level module or a new one if one is not present.
diff --git a/src/md/release.md b/src/md/release.md
index a5ad9b0..669a06a 100644
--- a/src/md/release.md
+++ b/src/md/release.md
@@ -152,8 +152,13 @@
## Build and Deploy Documentation
-The documentation will be generated as static HTML files and copied into the `apex-site` repository. It will be available at an alternate URL path until the release is promoted, at which time it will also be promoted to the main website location.
+The documentation will be generated as static HTML files into the `apex-site` repository, [separated by version (X.Y)](https://github.com/apache/apex-site/tree/asf-site/docs).
+**Note** You need Python 2.7+ and [mkdocs](http://www.mkdocs.org/) with patch for issue [mkdocs #859](https://github.com/mkdocs/mkdocs/issues/859) on top of the currently available version 0.15.3. After installing mkdocs with pip, run the following to obtain this build:
+
+```
+sudo pip install --upgrade git+https://github.com/sashadt/mkdocs.git
+```
Do the following setup steps before building and deploying the documentation.
1. Clone the apex-site repository into a folder called ```apex-site``` at the same level as the current repository.
@@ -176,10 +181,10 @@
The `REPO_NAME` variable above should match the folder name of the cloned apex module being built.
-To build and deploy the documentation, execute the following commands in the apex module folder specified in `REPO_NAME` above. **Note**: Until [mkdocs #859](https://github.com/mkdocs/mkdocs/issues/859) is resolved and available for download, use mkdocs built against [master](https://github.com/mkdocs/mkdocs).
+Build and deploy the documentation in the release directory:
```bash
-# build docs, they would be generated in a site sub-folder
+# build docs in ${REPO_NAME}, they will be generated in a site sub-folder
mkdocs build --clean
# Calculate the major.minor version
@@ -195,11 +200,7 @@
git push
```
-The documentation is now accessible at the URL location below. For patch releases, however, the documentation changes will already reflect on the website.
-
-```
-https://apex.apache.org/docs/${DOC_NAME}-${docv}/
-```
+After [publishing the site](https://github.com/apache/apex-site/tree/master#contributing) the new documentation will be available at `https://apex.apache.org/docs/${DOC_NAME}-${docv}/`
## Voting
diff --git a/src/md/verification.md b/src/md/verification.md
index eb394ea..47cd2c3 100644
--- a/src/md/verification.md
+++ b/src/md/verification.md
@@ -80,11 +80,11 @@
### Check for compilation, license headers, etc.
-For Apex:
+For Apex Core:
```bash
mvn clean apache-rat:check verify -Dlicense.skip=false install
```
-For Malhar:
+For Apex Malhar:
```bash
mvn clean apache-rat:check verify -Dlicense.skip=false -Pall-modules install
```
diff --git a/src/pages/index.html b/src/pages/index.html
index 27b3b30..48fdc48 100644
--- a/src/pages/index.html
+++ b/src/pages/index.html
@@ -21,6 +21,13 @@
</div>
<div class="break"></div>
+ <div class="container">
+ <center>
+ <a class="h2" href="https://blogs.apache.org/apex/entry/apache_apex_malhar_3_5">
+ <i>Now with <b>event-time windowing</b> and <b>high-level API.</b></i>
+ </a>
+ </center>
+ </div>
<div class="container">
<div class="col-md-4">
@@ -33,7 +40,7 @@
</div>
<div class="col-md-4">
<h2>Modular</h2>
- <p class="text-regular"><strong>The Apex platform comes with Malhar</strong>, a library of operators (modular units of business logic) that can be leveraged to quickly create new and non-trivial applications.</p>
+ <p class="text-regular"><strong>The Apex platform comes with Malhar</strong>, a library of operators (units of functionality) that can be leveraged to quickly create non-trivial applications. Includes many connectors for messaging systems, databases, files etc.</p>
</div>
</div>
