Apache Edgent is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
See README.md for high-level information about Apache Edgent.
This document describes building and the development of Apache Edgent itself, not how to develop Edgent applications.
The Edgent community welcomes contributions, please Get Involved!
If you are interested in developing a new connector see Writing Connectors for Edgent Applications
See the Edgent Wiki for additional information including Internal and Design notes.
See the updated Building and Using Eclipse sections below. The Ant and Gradle tooling is no longer functional.
It's recommended that developers of Edgent create a new workspace instead of reusing current gradle-based Edgent workspaces.
The develop branch is used for development. Jenkins is setup to build this branch and publish internal SNAPSHOT build results to the ASF Nexus SNAPSHOTS Repository (https://repository.apache.org/content/repositories/snapshots).
The master branch contains released code. Releases are published to the ASF Nexus Releases Repository (https://repository.apache.org/content/repositories/releases). The Releases repository is automatically mirrored to the Maven Central Repository.
See src/site/asciidoc/releasing.adoc for a more complete overview of the branches and their content. The Release Manager's Guide may also add some clarity (https://cwiki.apache.org/confluence/display/EDGENT/Release+Manager%27s+Guide)
Once you have forked the repository and created your local clone you need to download these additional development software tools.
toolchain supportMaven is used as build tool. Currently there are two options:
mvnw or mvnw.bat command - preferred)mvn command)The maven-wrapper will automatically download and install the correct Maven version and use that. Besides this, there is no difference between using the mvn and mvnw command.
You may also use a maven-integrated IDE for Edgent development. e.g., see the Using Eclipse section below.
All Edgent runtime development is done using Java 8. JARs for Java 7 and Android platforms are created by back-porting the compiled Java 8 code using a tool called retrolambda. More details on this below.
Per default the build will use Java 8 to perform the build of the Java 7 and Android modules. In order to reliably test the Java 7 modules on a real Java 7 Runtime, we defined an additional profile toolchain which lets Maven run the tests in the Java 7 Modules with a real Java 7 Runtime.
In preparation for testing the Java 7 and Android modules with enabled toolchain support, edit or create ~/.m2/toolchains.xml:
<?xml version="1.0" encoding="UTF8"?> <toolchains> <toolchain> <type>jdk</type> <provides> <version>1.8</version> <vendor>oracle</vendor> </provides> <configuration> <jdkHome>{path to the Java 8 SDK}</jdkHome> </configuration> </toolchain> <toolchain> <type>jdk</type> <provides> <version>1.7</version> <vendor>oracle</vendor> </provides> <configuration> <jdkHome>{path to the Java 7 SDK}</jdkHome> </configuration> </toolchain> <toolchains>
Set the jdkHome values appropriately for your system. e.g., on an OSX system:
j8 jdkHome: /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home j7 jdkHome: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home
Any pull request is expected to maintain the build success of mvn package.
To build and test for Java 8
$ ./mvnw clean package # -DskipTests to omit tests
To build and properly test the Edgent Java 7 and Android platform jars requires an installed Java 7 JRE and the toolchains.xml setup above, then
$ ./mvnw clean package -Djava8.home=$JAVA_HOME -Ptoolchain,platform-java7,platform-android
A set of Maven profiles have been created to control which parts should be built. The default profile only builds and tests the Java 8 versions of all modules and doesn‘t assemble a binary distribution as usually Maven builds don’t require such a step. It also doesn't build the Java 7 or Android modules either.
Edgent currently comes with these profiles:
apache-release: Builds source release bundle under targetdistribution: Builds one binary distribution bundle under distribution/target for Java 8. If the java 7 and android profiles are enabled too, for each of these an additional binary distribution is created.platform-java7: Builds Java 7 versions of all Edgent modules and runs the tests.platform-android: Builds Android versions of all Edgent modules that are compatible with Android (See JAVA_SUPPORT.md.toolchain: Runs the tests in the Java 7 and Android modules using a Java 7 runtime instead of Java 8 version, which happens if this profile is not enabled.As the Android modules are based on the Java 7 versions, when building the platform-android profile, the platform-java7 profile is required to be enabled too, or the build will fail.
For a not quite two hour introduction into Maven please feel free to watch this video we created for another Apache project: https://vimeo.com/167857327
Note: Apache Edgent releases include convenience binaries. Use of them is covered in samples/APPLICATION_DEVELOPMENT.md.
If instead you want to build Edgent for your use there are two different use-cases:
Build, test, and install the Edgent Java8 jars in the local maven repository
$ ./mvnw clean install # -DskipTests to skip tests
To also build, but not test, the Edgent Java 7 and Android platform jars:
$ ./mvnw clean install -DskipTests -Pplatform-java7,platform-android
To build and properly test the Edgent Java 7 and Android platform jars requires an installed Java 7 JRE and the toolchains.xml setup above, then
$ ./mvnw clean install -Djava8.home=$JAVA_HOME -Ptoolchain,platform-java7,platform-android
Build Edgent as described above to populate the local maven repository. Then see samples/APPLICATION_DEVELOPMENT.md for information about the get-edgent-jars.sh script.
An alternative to using the get-edgent-jars.sh script is to create a binary distribution bundle consisting of the Edgent runtime jars and their external dependencies.
Build a binary distribution bundle for Java 8
$ ./mvnw clean package -DskipTests -Pdistribution
The distribution bundle is created under distribution/target. The libs directory inside the bundle contains the Edgent jars and the ext directory contains third party dependencies the Edgent jars require.
NOTE: each third party dependency in the bundle comes with its own copyright and license terms which you implicitly accept when using a distribution bundle. See the README file in the bundle for more information.
You will need to manually setup the CLASSPATH for the build tooling that you're using to develop your Edgent application.
To also build the Edgent Java 7 and Android platform jars:
$ ./mvnw clean package -DskipTests -Pdistribution -Pplatform-java7,platform-android
The distribution bundles will be in platforms/java7/distribution/target and platforms/android/distribution/target respectively.
When a pull request is opened on the GitHub mirror site, the Travis CI service runs a full build of the java8 modules.
The latest build status for the project's branches can be seen at: https://travis-ci.org/apache/incubator-edgent/branches
The build setup is contained in .travis.yml in the project root directory. It includes:
In an attempt to more generally desentize tmo failures when the system property edgent.build.ci=true is set some runtime and test infrastructure components will bump the normal tmo value (e.g., 10x). This affects travis and Jenkins runs (both set edgent.build.ci). See:
* TStreamTest.waitForCompletion()
* AbstractTester.complete()
* Execuatble.invokeAction()
* generally search for uses of edgent.build.ci
* maybe remove other test specific uses of it in light of the general change
The following may now best be avoided: If your test randomly fails because, for example, it depends on publicly available test services, or is timing dependent, and if timing variances on the Travis CI servers may make it more likely for your tests to fail, you may disable the test from being executed on Travis CI using the following statement:
@Test public void testMyMethod() { assumeTrue(!Boolean.getBoolean("edgent.build.ci")); // your test code comes here ... }
Closing and reopening a pull request will kick off a new build against the pull request.
In addition to Travis CI running the quick tests with only the Java8 modules, we have also setup additional build-jobs at the Apaches Jenkins instance at https://builds.apache.org/view/E-G/view/Edgent/
This build also automatically runs on every commit, but in contrast to the Travis build, it also builds and tests the Java7 and Android modules using the toolchain profile.
This is also the build which produces and deploys the Maven artifacts that are published to the Apache Maven repository at https://repository.apache.org/
As an additional quality assurance tool, this build also runs a SonarQube analysis who's results are available at Apaches SonarQube instance at https://builds.apache.org/analysis/overview?id=45154
Heads up: the (Jenkins?) test failure reporting tooling seems to get confused in the face of the same named tests being run for multiple platforms. Generally you will see each test file listed twice: once for Java8 and once for Java7. In the html results it seems impossible to tell which platform a failed test (or passed test for that matter) applies to. Even though the html links for the two tests differ (e.g., the 2nd one has a “_2” at the end of the URL), a failed test‘s page shows the passed test’s page. My approach to investigating failures is to open the “View as plain text” page and then use the browser's search feature to look for the test name of interest to locate the output for the failing test. ugh.
Java 7 and Android target platforms are supported through use of retrolambda to convert Edgent Java 8 JARs to Java 7 JARs. In order to make it easy to address easily, for each Java 8 module a matching Java 7 version is located inside the <edgent>/platforms/java7 directory. For Android only those counterparts exist which are generally supported on Android.
In general all Java 7 modules differ from the ordinary Java 8 versions as these modules don't contain any source code or resources. They are all built by unpacking the Java 8 jars content into the current modules target directory. So the output is effectively located exactly in the same location it would have when normally compiling the Java 8 version. There the retrolambda plugin is executed to convert the existing class files into ones compatible with Java 7.
The Android versions are even simpler, as all they do is unpack the Java 7 versions and re-pack the content with the android groupId. All except the two modules which are currently only available on Android (located in the <edgent>/platforms/android/android directory). These modules are built up similar to the Java 8 versions, but they also contain the retrolambda plugin execution. While it would have been possible to treat these modules as Java 7, for the sake of an equal coding experience it was decided to make it possible to write the same type of code for all modules.
An Android module's dependency on the Java 7 version makes the requirement obvious, that in order to build the Android versions, the Java 7 versions have to be built too.
See JAVA_SUPPORT.md for which Edgent capabilities / JARs are supported for each environment.
Also see Coding Conventions below.
The typical maven build contains two phases of unit-tests. The Unit-Test phase which is executed by the surefire maven plugin and the Integration-Test phase, which is executed by the failsafe maven plugin.
When running a normal maven package build, only the unit-test phase is executed. When running verify or above (install, deploy, etc.) the integration tests are also executed.
Each Maven plugin produces output to different directories:
<module>/target/surefire-reports - JUnit unit-test reports<module>/target/failsafe-reports - JUnit integration-test reportsIn addition to running the unit tests, coverage data is automatically collected by the jacoco-maven-plugin, which is configured to store its data in <module>/target/coverage-reports in files called jacoco-ut.exec and jacoco-it.exec.
Even if at least the surfire and failsafe output is generated in a human readable txt and xml form, the jacoco output is intended on being used by tools. SonarQube is for example able to interpret this information In order to generate nicely formatted html reports, please have a look at the following Site generation chapter.
Maven has 3 built in lifecycles:
If the human readable version of all of these should be generated, all needed to do this, is to append a simple site:site at the end of the maven command.
./mvnw -Pdistribution,platform-java7,platform-android clean verify site:site
Each modules <module>/target/site directory will then contain the generated Module documentation.
Javadoc generation is a bit of a mess / problem.
Our customized “aggregated javadoc” (as seen on the website) is formed by configurations of the maven-javadoc-plugin. As noted in the pom, that configuration has no effect for mvn javadoc:aggregate, only mvn site.
Then, as also noted in the pom, a problem was encountered in the release processing so the aggregate reportSet was commented out. Hence only non-aggregated javadoc is created.
Until someone figures out how to address this in a reasonable manner, the following can be used:
# edit pom.xml to uncomment the javadoc plugin config's aggregate reportSet ./mvnw install -DskipTests # "package" isn't enough ./mvnw site # target/site/apidocs will have the aggregated javadoc
To generate this javadoc for a release, in the clone you‘re doing the release in, and after the release:perform, cd to the release’s target/checkout directory and perform the above cmds
Or, create a new clone from the release's tag (git clone --branch edgent-<ver> the-https-repo-url tmp-javadoc-clone) and do the cmds there.
Note, trying to do the above using a source-release tar/zip bundle fails because mvn site must be run in a git repo.
There is a lot of surface area to the maven build tooling. The following information may help to better understand it.
pom.xml/maven-surefile-plugin - unit test executionpom.xml/maven-failsafe-plugin - integration test executionpom.xml/jacoco-maven-plugin - jacoco code coverage reportspom.xml/animal-sniffer-maven-plugin - retrolambda results checkerpom.xml/org.codehaus.sonar-plugins - SonarQube code quality reportspom.xml/maven-javadoc-plugin - javadoc generation and the config for all of the “grouping” control.pom.xml/apache-rat-plugin - builds automatically run Apache RAT (Release Audit Tool) for checking for appropriate content.pom.xml/maven-assembly-plugin - used in a couple places for configuring and generating “assemblies” => source release bundle, distribution bundlespom.xml/maven-site-plugin - things related to website generation that includes a number of interesting things, including html reports from various things above plus aggregated javadoc. How / if this untimately relates to the public website is TDB.platforms/java7/pom.xml/retrolambda-maven-plugin and platforms/android/android/pom.xml/retrolambda-maven-plugin - where retrolambda is enabledpom.xml/maven-remote-resources-plugin config plays a role in all of thissrc/main/appended-resources - contains copies of license text for artifacts that are bundled in Edgent bundles. For the most part this means the Edgent Console jar/war. These are incorporated into a jar by including a declaration like the following in the project's pom:<resource> <directory>${project.basedir}/../../src/main/appended-resources/licenses</directory> <targetPath>${project.build.directory}/${project.artifactId}-${project.version}/META-INF/licenses</targetPath> </resource>
* `src/main/ibm-remote-resources` - contains the NOTICE fragment for projects containing IBM contributed code. Applicable project's define the following in their pom to ensure a correct NOTICE is included in the project's jar:
<properties> <remote-resources-maven-plugin.remote-resources.dir>../../src/main/ibm-remote-resources</remote-resources-maven-plugin.remote-resources.dir> </properties>
* `edgent-console-servlets:war` contains bundled code (downloaded and incorporated by the build tooling). It includes the bundled code's license text as described above. Its own LICENSE and NOTICE is copied from the respective files in its `src/main/remote-resources/META-INF`. <b>There are copies of those in under the java7 platform as well.</b> * `edgent-console-server:jar` bundles the console-servlets war and as such requires the same LICENSE/NOTICE/licenses treatment as the servlets war. <b>There are copies of its LICENSE/NOTICE in its `src/main/remote-resources/META-INF`. There are copies of those in under the java7 platform as well.</b>
src/assembly/source-release.xml - configuration information controlling source-release bundle and distribution bundle names, included/excluded files, etc.samples/get-edgent-jars-project uses the same scheme and has its own copies of the files.utils/edgent-deployment-filter-maven-plugin - a plugin for eliminating the publishing of test related jars during a release. See EDGENT-440. The plugin is built and released separately.The kafka connector tests aren't run by default as the connector must connect to a running Kafka/Zookeeper config.
There are apparently ways to embed Kafka and Zookeeper for testing purposes but we're not there yet. Contributions welcome!
Setting up the servers is easy. Follow the steps in the KafkaStreamsTestManual javadoc.
Once kafka/zookeeper are running you can run the tests and samples:
#### run the kafka tests ./mvnw -pl connectors/kafka test '-Dtest=**/*Manual' # hint: to run just a single test case: '-Dtest=**/KafkaStreamsTestManual#testSimple' #### run the sample (cd samples; ./mvnw package -DskipTests) # build if not already done cd samples/connectors/scripts/kafka cat README ./runkafkasample.sh sub ./runkafkasample.sh pub
The code is broken into a number of projects and modules within those projects defined by directories under edgent. Each top level directory is a project and contains one or more modules:
api - The APIs for Edgent. In general there is a strict split between APIs and implementations to allow multiple implementations of an API, such as for different device types or different approaches.spi - Common implementation code that may be shared by multiple implementations of an API. There is no requirement for an API implementation to use the provided spi code.runtime - Implementations of APIs for executing Edgent applications at runtime. Initially a single runtime is provided, etiao - EveryThing Is An Oplet - A micro-kernel that executes Edgent applications by being a very simple runtime where all functionality is provided as oplets, execution objects that process streaming data. So an Edgent application becomes a graph of connected oplets, and items such as fan-in or fan-out, metrics etc. are implemented by inserting additional oplets into the application's graph.providers - Providers bring the Edgent modules together to allow Edgent applications to be developed and run.connectors - Connectors to files, HTTP, MQTT, Kafka, JDBC, etc. Connectors are modular so that deployed applications need only include the connectors they use, such as only MQTT. Edgent applications running at the edge are expected to connect to back-end systems through some form of message-hub, such as an MQTT broker, Apache Kafka, a cloud based IoT service, etc.apps - Applications for use in an Internet of Things environment.analytics - Analytics for use by Edgent applications.utils - Optional utilities for Edgent applications.console - Development console that allows visualization of the streams within an Edgent application during development.android - Code specific to Android.test - SVTSamples are located at https://github.com/apache/incubator-edgent-samples
Placeholder: see EDGENT-23
A couple of key items in the mean time:
rat to check license headers and fails if non-conforming files are encountered.Edgent‘s primary development environment is Java 8, to take advantage of lambda expressions since Edgent’s primary API is a functional one.
However, in order to support Android (and Java 7), other features of Java 8 are not used in the core code. Lambdas are translated into Java 7 compatible classes using retrolambda.
Thus for core code and tests that needs to run on Android/Java7:
In general, most code is expected to work on Android (but might not yet) with the exception of these excluded features:
SLF4J is used for logging and tracing.
Search the code for org.slf4j.LoggerFactory to see a sample of its use.
The Edgent code is in ASF resident git repositories:
https://gitbox.apache.org/repos/asf/incubator-edgent.git
The repositories are mirrored on GitHub:
https://github.com/apache/incubator-edgent
Use of the normal GitHub workflow brings benefits to the team including lightweight code reviewing, automatic regression tests, etc. for both committers and non-committers.
For a description of the GitHub workflow, see:
https://guides.github.com/introduction/flow/ https://guides.github.com/activities/hello-world/
In summary:
Since the GitHub incubator-edgent repository is a mirror of the ASF repository, the usual GitHub based merge workflow for committers isn’t supported.
Committers can use one of several ways to ultimately merge the pull request into the repo at the ASF. One way is described here:
Notes with the above PR merge directions:
$ git remote add mirror https://gitbox.apache.org/repos/asf/incubator-edgent.gitThe Edgent Git repository, or source release bundle, contains Maven project definitions for the various components of Edgent such as api, runtime, connectors.
Once you import the Maven projects into your workspace, builds and JUnit testing of Edgent in Eclipse use the same artifacts as the Maven command line tooling. Like the command line tooling, the jars for dependent projects are automatically downloaded to the local maven repository and used.
If you want to use Eclipse to clone your fork, use the Eclipse Git Team Provider plugin
git@github.com:<username>/incubator-edgent.git). The other fields will be populated automatically. Click Next. If required, enter your passphrase.master) and click NextOnce you have cloned the Git repository to your machine or are working from an unpacked source release bundle, import the Maven projects into your workspace
Top-level artifacts such as README.md are available under the edgent-parent project.
Note: Specifics may change depending on your version of Eclipse or the Eclipse Maven or Git Team Provider.
The ALv2 license headers in various markdown files (e.g., README.md) seem to confuse the Eclipse wikitext editor resulting in blank contents in its preview panel. This situation may be improved by installing the Markdown text editor from the Eclipse marketplace and adjusting Eclipse's file associations accordingly.
Apache Edgent is the new name and the conversion is complete.
Code changes:
Users of Edgent will need to update their references to the above. It's recommended that developers of Edgent create a new workspace instead of reusing their Quarks workspace.