Apache Edgent is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
See README.md for high-level information about Apache Edgent.
This document describes building and the development of Apache Edgent itself, not how to develop Edgent applications.
The Edgent community welcomes contributions, please Get Involved!
If you are interested in developing a new connector see Writing Connectors for Edgent Applications
See the Edgent Wiki for additional information including Internal and Design notes.
See the updated Building and Using Eclipse sections below. The Ant and Gradle tooling is no longer functional.
It's recommended that developers of Edgent create a new workspace instead of reusing current gradle-based Edgent workspaces.
The develop
branch is used for development. Jenkins is setup to build this branch and publish internal SNAPSHOT build results to the ASF Nexus SNAPSHOTS Repository (https://repository.apache.org/content/repositories/snapshots).
The master
branch contains released code. Releases are published to the ASF Nexus Releases Repository (https://repository.apache.org/content/repositories/releases). The Releases repository is automatically mirrored to the Maven Central Repository.
Once you have forked the repository and created your local clone you need to download these additional development software tools.
toolchain
supportMaven is used as build tool. Currently there are two options:
mvnw
or mvnw.bat
command - preferred)mvn
command)The maven-wrapper will automatically download and install the correct Maven version and use that. Besides this, there is no difference between using the mvn
and mvnw
command.
You may also use a maven-integrated IDE for Edgent development. e.g., see the Using Eclipse section below.
All Edgent runtime development is done using Java 8. JARs for Java 7 and Android platforms are created by back-porting the compiled Java 8 code using a tool called retrolambda
. More details on this below.
Per default the build will use Java 8 to perform the build of the Java 7 and Android modules. In order to reliably test the Java 7 modules on a real Java 7 Runtime, we defined an additional profile toolchain
which lets Maven run the tests in the Java 7 Modules with a real Java 7 Runtime.
In preparation for testing the Java 7 and Android modules with enabled toolchain
support, edit or create ~/.m2/toolchains.xml
:
<?xml version="1.0" encoding="UTF8"?> <toolchains> <toolchain> <type>jdk</type> <provides> <version>1.8</version> <vendor>oracle</vendor> </provides> <configuration> <jdkHome>{path to the Java 8 SDK}</jdkHome> </configuration> </toolchain> <toolchain> <type>jdk</type> <provides> <version>1.7</version> <vendor>oracle</vendor> </provides> <configuration> <jdkHome>{path to the Java 7 SDK}</jdkHome> </configuration> </toolchain> <toolchains>
Set the jdkHome values appropriately for your system. e.g., on an OSX system:
j8 jdkHome: /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home j7 jdkHome: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home
Any pull request is expected to maintain the build success of mvn package
.
To build and test for Java 8
$ ./mvnw clean package # -DskipTests to omit tests
To build and properly test the Edgent Java 7 and Android platform jars requires an installed Java 7 JRE and the toolchains.xml
setup above, then
$ ./mvnw clean package -Djava8.home=$JAVA_HOME -Ptoolchain,platform-java7,platform-android
A set of Maven profiles
have been created to control which parts should be built. The default profile only builds and tests the Java 8 versions of all modules and doesn‘t assemble a binary distribution as usually Maven builds don’t require such a step. It also doesn't build the Java 7 or Android modules either.
Edgent currently comes with these profiles:
apache-release
: Builds source release bundle under target
distribution
: Builds one binary distribution bundle under distribution/target
for Java 8. If the java 7 and android profiles are enabled too, for each of these an additional binary distribution is created.platform-java7
: Builds Java 7 versions of all Edgent modules and runs the tests.platform-android
: Builds Android versions of all Edgent modules that are compatible with Android (See JAVA_SUPPORT.md.toolchain
: Runs the tests in the Java 7 and Android modules using a Java 7 runtime instead of Java 8 version, which happens if this profile is not enabled.As the Android modules are based on the Java 7 versions, when building the platform-android
profile, the platform-java7
profile is required to be enabled too, or the build will fail.
For a not quite two hour introduction into Maven please feel free to watch this video we created for another Apache project: https://vimeo.com/167857327
Note: Apache Edgent releases include convenience binaries. Use of them is covered in samples/APPLICATION_DEVELOPMENT.md.
If instead you want to build Edgent for your use there are two different use-cases:
Build, test, and install the Edgent Java8 jars in the local maven repository
$ ./mvnw clean install # -DskipTests to skip tests
To also build, but not test, the Edgent Java 7 and Android platform jars:
$ ./mvnw clean install -DskipTests -Pplatform-java7,platform-android
To build and properly test the Edgent Java 7 and Android platform jars requires an installed Java 7 JRE and the toolchains.xml
setup above, then
$ ./mvnw clean install -Djava8.home=$JAVA_HOME -Ptoolchain,platform-java7,platform-android
Build Edgent as described above to populate the local maven repository. Then see samples/APPLICATION_DEVELOPMENT.md for information about the get-edgent-jars.sh
script.
An alternative to using the get-edgent-jars.sh
script is to create a binary distribution bundle consisting of the Edgent runtime jars and their external dependencies.
Build a binary distribution bundle for Java 8
$ ./mvnw clean package -DskipTests -Pdistribution
The distribution bundle is created under distribution/target
. The libs
directory inside the bundle contains the Edgent jars and the ext
directory contains third party dependencies the Edgent jars require.
NOTE: each third party dependency in the bundle comes with its own copyright and license terms which you implicitly accept when using a distribution bundle. See the README file in the bundle for more information.
You will need to manually setup the CLASSPATH for the build tooling that you're using to develop your Edgent application.
To also build the Edgent Java 7 and Android platform jars:
$ ./mvnw clean package -DskipTests -Pdistribution -Pplatform-java7,platform-android
The distribution bundles will be in platforms/java7/distribution/target
and platforms/android/distribution/target
respectively.
When a pull request is opened on the GitHub mirror site, the Travis CI service runs a full build of the java8 modules.
The latest build status for the project's branches can be seen at: https://travis-ci.org/apache/incubator-edgent/branches
The build setup is contained in .travis.yml
in the project root directory. It includes:
In an attempt to more generally desentize tmo failures when the system property edgent.build.ci=true is set some runtime and test infrastructure components will bump the normal tmo value (e.g., 10x). This affects travis and Jenkins runs (both set edgent.build.ci). See:
* TStreamTest.waitForCompletion() * AbstractTester.complete() * Execuatble.invokeAction() * generally search for uses of edgent.build.ci * maybe remove other test specific uses of it in light of the general change
The following may now best be avoided: If your test randomly fails because, for example, it depends on publicly available test services, or is timing dependent, and if timing variances on the Travis CI servers may make it more likely for your tests to fail, you may disable the test from being executed on Travis CI using the following statement:
@Test public void testMyMethod() { assumeTrue(!Boolean.getBoolean("edgent.build.ci")); // your test code comes here ... }
Closing and reopening a pull request will kick off a new build against the pull request.
In addition to Travis CI running the quick tests with only the Java8 modules, we have also setup additional build-jobs at the Apaches Jenkins instance at https://builds.apache.org/view/E-G/view/Edgent/
This build also automatically runs on every commit, but in contrast to the Travis build, it also builds and tests the Java7 and Android modules using the toolchain profile.
This is also the build which produces and deploys the Maven artifacts that are published to the Apache Maven repository at https://repository.apache.org/
As an additional quality assurance tool, this build also runs a SonarQube analysis who's results are available at Apaches SonarQube instance at https://builds.apache.org/analysis/overview?id=45154
Heads up: the (Jenkins?) test failure reporting tooling seems to get confused in the face of the same named tests being run for multiple platforms. Generally you will see each test file listed twice: once for Java8 and once for Java7. In the html results it seems impossible to tell which platform a failed test (or passed test for that matter) applies to. Even though the html links for the two tests differ (e.g., the 2nd one has a “_2” at the end of the URL), a failed test‘s page shows the passed test’s page. My approach to investigating failures is to open the “View as plain text” page and then use the browser's search feature to look for the test name of interest to locate the output for the failing test. ugh.
Java 7 and Android target platforms are supported through use of retrolambda to convert Edgent Java 8 JARs to Java 7 JARs. In order to make it easy to address easily, for each Java 8 module a matching Java 7 version is located inside the <edgent>/platforms/java7
directory. For Android only those counterparts exist which are generally supported on Android.
In general all Java 7 modules differ from the ordinary Java 8 versions as these modules don't contain any source code or resources. They are all built by unpacking the Java 8 jars content into the current modules target directory. So the output is effectively located exactly in the same location it would have when normally compiling the Java 8 version. There the retrolambda plugin is executed to convert the existing class files into ones compatible with Java 7.
The Android versions are even simpler, as all they do is unpack the Java 7 versions and re-pack the content with the android groupId. All except the two modules which are currently only available on Android (located in the <edgent>/platforms/android/android
directory). These modules are built up similar to the Java 8 versions, but they also contain the retrolambda plugin execution. While it would have been possible to treat these modules as Java 7, for the sake of an equal coding experience it was decided to make it possible to write the same type of code for all modules.
An Android module's dependency on the Java 7 version makes the requirement obvious, that in order to build the Android versions, the Java 7 versions have to be built too.
See JAVA_SUPPORT.md for which Edgent capabilities / JARs are supported for each environment.
Also see Coding Conventions below.
The typical maven build contains two phases of unit-tests. The Unit-Test phase which is executed by the surefire maven plugin and the Integration-Test phase, which is executed by the failsafe maven plugin.
When running a normal maven package
build, only the unit-test phase is executed. When running verify
or above (install
, deploy
, etc.) the integration tests are also executed.
Each Maven plugin produces output to different directories:
<module>/target/surefire-reports
- JUnit unit-test reports<module>/target/failsafe-reports
- JUnit integration-test reportsIn addition to running the unit tests, coverage data is automatically collected by the jacoco-maven-plugin
, which is configured to store its data in <module>/target/coverage-reports
in files called jacoco-ut.exec
and jacoco-it.exec
.
Even if at least the surfire and failsafe output is generated in a human readable txt and xml form, the jacoco output is intended on being used by tools. SonarQube is for example able to interpret this information In order to generate nicely formatted html reports, please have a look at the following Site generation
chapter.
Maven has 3 built in lifecycles:
If the human readable version of all of these should be generated, all needed to do this, is to append a simple site:site
at the end of the maven command.
./mvnw -Pdistribution,platform-java7,platform-android clean verify site:site
Each modules <module>/target/site
directory will then contain the generated Module documentation.
There is a lot of surface area to the maven build tooling. The following information may help to better understand it.
pom.xml/maven-surefile-plugin
- unit test executionpom.xml/maven-failsafe-plugin
- integration test executionpom.xml/jacoco-maven-plugin
- jacoco code coverage reportspom.xml/animal-sniffer-maven-plugin
- retrolambda results checkerpom.xml/org.codehaus.sonar-plugins
- SonarQube code quality reportspom.xml/maven-javadoc-plugin
- javadoc generation and the config for all of the “grouping” control.pom.xml/apache-rat-plugin
- builds automatically run Apache RAT (Release Audit Tool) for checking for appropriate content.pom.xml/maven-assembly-plugin
- used in a couple places for configuring and generating “assemblies” => source release bundle, distribution bundlespom.xml/maven-site-plugin
- things related to website generation that includes a number of interesting things, including html reports from various things above plus aggregated javadoc. How / if this untimately relates to the public website is TDB.platforms/java7/pom.xml/retrolambda-maven-plugin
and platforms/android/android/pom.xml/retrolambda-maven-plugin
- where retrolambda is enabledpom.xml/maven-remote-resources-plugin
config plays a role in all of thissrc/main/appended-resources
- contains copies of license text for artifacts that are bundled in Edgent bundles. For the most part this means the Edgent Console jar/war. These are incorporated into a jar by including a declaration like the following in the project's pom:<resource> <directory>${project.basedir}/../../src/main/appended-resources/licenses</directory> <targetPath>${project.build.directory}/${project.artifactId}-${project.version}/META-INF/licenses</targetPath> </resource>
* `src/main/ibm-remote-resources` - contains the NOTICE fragment for projects containing IBM contributed code. Applicable project's define the following in their pom to ensure a correct NOTICE is included in the project's jar:
<properties> <remote-resources-maven-plugin.remote-resources.dir>../../src/main/ibm-remote-resources</remote-resources-maven-plugin.remote-resources.dir> </properties>
* `edgent-console-servlets:war` contains bundled code (downloaded and incorporated by the build tooling). It includes the bundled code's license text as described above. Its own LICENSE and NOTICE is copied from the respective files in its `src/main/remote-resources/META-INF`. <b>There are copies of those in under the java7 platform as well.</b> * `edgent-console-server:jar` bundles the console-servlets war and as such requires the same LICENSE/NOTICE/licenses treatment as the servlets war. <b>There are copies of its LICENSE/NOTICE in its `src/main/remote-resources/META-INF`. There are copies of those in under the java7 platform as well.</b>
src/assembly/source-release.xml
- configuration information controlling source-release bundle and distribution bundle names, included/excluded files, etc.samples/get-edgent-jars-project
uses the same scheme and has its own copies of the files.The kafka connector tests aren't run by default as the connector must connect to a running Kafka/Zookeeper config.
There are apparently ways to embed Kafka and Zookeeper for testing purposes but we're not there yet. Contributions welcome!
Setting up the servers is easy. Follow the steps in the KafkaStreamsTestManual javadoc.
Once kafka/zookeeper are running you can run the tests and samples:
#### run the kafka tests ./mvnw -pl connectors/kafka test '-Dtest=**/*Manual' #### run the sample (cd samples; ./mvnw package -DskipTests) # build if not already done cd samples/scripts/connectors/kafka cat README ./runkafkasample.sh sub ./runkafkasample.sh pub
The code is broken into a number of projects and modules within those projects defined by directories under edgent
. Each top level directory is a project and contains one or more modules:
api
- The APIs for Edgent. In general there is a strict split between APIs and implementations to allow multiple implementations of an API, such as for different device types or different approaches.spi
- Common implementation code that may be shared by multiple implementations of an API. There is no requirement for an API implementation to use the provided spi code.runtime
- Implementations of APIs for executing Edgent applications at runtime. Initially a single runtime is provided, etiao
- EveryThing Is An Oplet - A micro-kernel that executes Edgent applications by being a very simple runtime where all functionality is provided as oplets, execution objects that process streaming data. So an Edgent application becomes a graph of connected oplets, and items such as fan-in or fan-out, metrics etc. are implemented by inserting additional oplets into the application's graph.providers
- Providers bring the Edgent modules together to allow Edgent applications to be developed and run.connectors
- Connectors to files, HTTP, MQTT, Kafka, JDBC, etc. Connectors are modular so that deployed applications need only include the connectors they use, such as only MQTT. Edgent applications running at the edge are expected to connect to back-end systems through some form of message-hub, such as an MQTT broker, Apache Kafka, a cloud based IoT service, etc.apps
- Applications for use in an Internet of Things environment.analytics
- Analytics for use by Edgent applications.utils
- Optional utilities for Edgent applications.console
- Development console that allows visualization of the streams within an Edgent application during development.android
- Code specific to Android.test
- SVTSamples are located at https://github.com/apache/incubator-edgent-samples
Placeholder: see EDGENT-23
A couple of key items in the mean time:
rat
to check license headers and fails if non-conforming files are encountered.Edgent‘s primary development environment is Java 8, to take advantage of lambda expressions since Edgent’s primary API is a functional one.
However, in order to support Android (and Java 7), other features of Java 8 are not used in the core code. Lambdas are translated into Java 7 compatible classes using retrolambda.
Thus for core code and tests that needs to run on Android/Java7:
In general, most code is expected to work on Android (but might not yet) with the exception of these excluded features:
SLF4J is used for logging and tracing.
Search the code for org.slf4j.LoggerFactory to see a sample of its use.
The Edgent code is in ASF resident git repositories:
https://git-wip-us.apache.org/repos/asf/incubator-edgent.git
The repositories are mirrored on GitHub:
https://github.com/apache/incubator-edgent
Use of the normal GitHub workflow brings benefits to the team including lightweight code reviewing, automatic regression tests, etc. for both committers and non-committers.
For a description of the GitHub workflow, see:
https://guides.github.com/introduction/flow/ https://guides.github.com/activities/hello-world/
In summary:
Since the GitHub incubator-edgent repository is a mirror of the ASF repository, the usual GitHub based merge workflow for committers isn’t supported.
Committers can use one of several ways to ultimately merge the pull request into the repo at the ASF. One way is described here:
Notes with the above PR merge directions:
$ git remote add mirror https://github.com/apache/incubator-edgent.git
The Edgent Git repository, or source release bundle, contains Maven project definitions for the various components of Edgent such as api, runtime, connectors.
Once you import the Maven projects into your workspace, builds and JUnit testing of Edgent in Eclipse use the same artifacts as the Maven command line tooling. Like the command line tooling, the jars for dependent projects are automatically downloaded to the local maven repository and used.
If you want to use Eclipse to clone your fork, use the Eclipse Git Team Provider plugin
git@github.com:<username>/incubator-edgent.git
). The other fields will be populated automatically. Click Next. If required, enter your passphrase.master
) and click NextOnce you have cloned the Git repository to your machine or are working from an unpacked source release bundle, import the Maven projects into your workspace
Top-level artifacts such as README.md
are available under the edgent-parent
project.
Note: Specifics may change depending on your version of Eclipse or the Eclipse Maven or Git Team Provider.
The ALv2 license headers in various markdown files (e.g., README.md) seem to confuse the Eclipse wikitext
editor resulting in blank contents in its preview panel. This situation may be improved by installing the Markdown text editor
from the Eclipse marketplace and adjusting Eclipse's file associations accordingly.
Apache Edgent is the new name and the conversion is complete.
Code changes:
Users of Edgent will need to update their references to the above. It's recommended that developers of Edgent create a new workspace instead of reusing their Quarks workspace.