Apache Edgent is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
See README.md for high-level information about Apache Edgent.
This document describes development of Apache Edgent itself, not how to develop Edgent applications.
The Edgent community welcomes contributions, please Get Involved!
If you are interested in developing a new connector see Writing Connectors for Edgent Applications
See the Edgent Wiki for additional information including Internal and Design notes.
See the updated Building and Using Eclipse sections below. The Ant tooling is no longer functional.
It's recommended that developers of Edgent create a new workspace instead of reusing current ant-based Edgent workspaces.
Apache Edgent is the new name and the conversion is complete.
Code changes:
Users of Edgent will need to update their references to the above. It's recommended that developers of Edgent create a new workspace instead of reusing their Quarks workspace.
Once you have forked the repository and created your local clone you need to download these additional development software tools.
All Edgent runtime development is done using Java 8. JARs for Java 7 and Android platforms are created as described below.
Building from a source release bundle (lacking a ./gradlew
) requires performing a one-time bootstrap step using an installed version of gradle:
$ gradle # one time gradle build bootstrap setup.
Building an Edgent binary release bundle:
$ ./gradlew release
The build reports the location of the binary distribution bundle that can then be unpacked and used in building applications.
See Getting Started for information on using the binary release bundle.
The primary build process is using Gradle, any pull request is expected to maintain the build success of clean, assemble, test
.
The Gradle wrapper edgent/{gradlew,gradlew.bat}
should be used. The wrapper ensures the appropriate version of Gradle is used and it will automatically download it if needed, e.g.:
$ ./gradlew --version $ ./gradlew clean build
The Gradle tooling:
<edgent>/build/release-edgent
<edgent>/build/distributions
and <edgent>/<project>/build
The top-level Gradle file is edgent/build.gradle
and it contains several unique tasks:
wrapper
(default) : one-time bootstrap processing for use when building from a source release bundleassemble
: Build all code and Javadoc into build\distributions
. The build will fail on any code error or Javadoc warning or error.all
: Synonym for assemble
build
: Essentially like “assemble test reports”clean
: Clean the projecttest
: Run the JUnit tests, if any test fails the test run stops. Use --continue
to not stop on the first failure.--tests
option to run a subset of the tests. Multiple --tests
options may be specified following each test task.$ ./gradlew <project>:test
$ ./gradlew <project>:test --tests '*.SomeTest'
$ ./gradlew <project>:test --tests '*.SomeTest.someMethod'
cleanTest
task to force rerunning a previously successful test task (without forcing a rerun of all of the task's dependencies):$ ./gradlew [<project>:]cleanTest [<project>:]test
reports
: Generate JUnit and Code Coverage reports in build\distributions\reports
. Use after executing the test
target.reports\tests\overview-summary.html
- JUnit test reportreports\coverage\index.html
- Code coverage reportrelease
: Build release bundles in build/release-edgent
, that includes subsets of the Edgent JARs that run on Java 7 (build/distributions/java7
) and Android (build/distributions/android
). By default, SNAPSHOT bundles are created. Specify -Dedgent.snapshotId=""
to create bundles for a formal release.rat
: run the Apache Release Analysis Tool (license checking).signAll
: Sign the release bundles in build/release-edgent
(first run release
). You will be promoted for your PGP code signing key's ID, the location of the keyring file, and the secret key password. Default response values may be set with environment variables:GPG_ID
- the code signing key's ID (e.g., D0F56CAD)GPG_SECRING
- path to the secret key's keyring fileThe build process has been tested on Linux and macOS.
To build on Windows probably needs some changes, please get involved and contribute them!
When a pull request is opened on the GitHub mirror site, the Travis CI service runs a full build.
The latest build status for the project's branches can be seen at: https://travis-ci.org/apache/incubator-edgent/branches
The build setup is contained in .travis.yml
in the project root directory. It includes:
If your test randomly fails because, for example, it depends on publicly available test services, or is timing dependent, and if timing variances on the Travis CI servers may make it more likely for your tests to fail, you may disable the test from being executed on Travis CI using the following statement:
@Test public void testMyMethod() { assumeTrue(!Boolean.getBoolean("edgent.build.ci")); // your test code comes here ... }
Closing and reopening a pull request will kick off a new build against the pull request.
Java 7 and Android target platforms are supported through use of retrolambda to convert Edgent Java8 JARs to Java7 JARs.
Building a release (./gradlew release
) produces three sets of JARs under
See JAVA_SUPPORT.md for which Edgent capabilities / JARs are supported for each environment.
The Gradle tooling uses some Ant tooling to create the Java 7 and Android platform JARs.
Java 7 Edgent runtime JARs are created using platform/java7/build.xml
. Adding a JAR just requires:
retro7.edgent
- Copy entry for an existing JAR.test7.setup
and test7.run
- Copy entry for an existing JAR.Any Java 7 JAR is automatically included in Android unless it is explictly excluded in platform/android/build.xml
.
Running the reports
target produces two reports:
builds/distributions/reports/tests/index.html
- JUnit test reportbuilds/distributions/reports/coverage/index.html
- Code coverage report.The kafka connector tests aren't run by default as the connector must connect to a running Kafka/Zookeeper config.
There are apparently ways to embedd Kafka and Zookeeper for testing purposes but we're not there yet. Contributions welcome!
Setting up the servers is easy. Follow the steps in the KafkaStreamsTestManual javadoc.
Once kafka/zookeeper are running you can run the tests and samples:
#### run the kafka tests ./gradlew connectors:kafka:test --tests '*.*Manual' #### run the sample cd java8/scripts/connectors/kafka cat README ./runkafkasample.sh sub ./runkafkasample.sh pub
The JDBC connector tests are written to run against Apache Derby as the backing dbms and the derby jar needs to be on the classpath. The tests are skipped if derby can't be loaded. The test harness adds $DERBY_HOME/db/lib/derby.jar to the classpath. See JdbcStreamsTest for more info but the following should suffice:
export DERBY_HOME=$JAVA_HOME/db #### if JAVA_HOME isn't set - e.g., on OSX... export DERBY_HOME=`/usr/libexec/java_home`/db
Once DERBY_HOME is set the tests and samples can be run as follows
#### run the jdbc tests ./gradlew connectors:jdbc:test #### run the sample cd java8/scripts/connectors/jdbc cat README ./runjdbcsample.sh writer ./runjdbcsample.sh reader
All of the standard build system tasks above must be run with JAVA_HOME
set to use a Java8 VM.
As noted above, the release
task includes generation of Java7 compatible versions of the Edgent JARs. After the release task has been run, Edgent may be tested in a Java7 context using some special test7 tasks.
See JAVA_SUPPORT for information about what Edgent features are supported in the different environments.
# pave the way for useful report generation at the end $ ./gradlew cleanTest $ ./gradlew reports # run with JAVA_HOME/PATH set for Java8 $ ./gradlew test7Compile # compile the Edgent tests to operate in a Java7 environment $ sh # muck with EVs for Java7 in a subshell $ export JAVA_HOME=`/usr/libexec/java_home -v 1.7` # on OSX $ export PATH=$JAVA_HOME/bin:$PATH $ ./gradlew test7Run # run the tests with a Java7 VM $ exit # run with JAVA_HOME set for Java8 $ ./gradlew test7Reports # generate the JUnit and coverage reports
Initial support for publishing to a local Maven repository has been added. Use the following to do the publish.
$ ./gradlew publishToMavenLocal
The component JARs / WARs are published as well as their sources. The published groupId is org.apache.edgent
. The artifactIds match the names of the JARs in the target-dir / release tgz.
For example: org.apache.edgent:edgent.api.topology:0.4.0
The code is broken into a number of projects and modules within those projects defined by directories under edgent
. Each top level directory is a project and contains one or more modules:
api
- The APIs for Edgent. In general there is a strict split between APIs and implementations to allow multiple implementations of an API, such as for different device types or different approaches.spi
- Common implementation code that may be shared by multiple implementations of an API. There is no requirement for an API implementation to use the provided spi code.runtime
- Implementations of APIs for executing Edgent applications at runtime. Initially a single runtime is provided, etiao
- EveryThing Is An Oplet - A micro-kernel that executes Edgent applications by being a very simple runtime where all functionality is provided as oplets, execution objects that process streaming data. So an Edgent application becomes a graph of connected oplets, and items such as fan-in or fan-out, metrics etc. are implemented by inserting additional oplets into the application's graph.providers
- Providers bring the Edgent modules together to allow Edgent applications to be developed and run.connectors
- Connectors to files, HTTP, MQTT, Kafka, JDBC, etc. Connectors are modular so that deployed applications need only include the connectors they use, such as only MQTT. Edgent applications running at the edge are expected to connect to back-end systems through some form of message-hub, such as an MQTT broker, Apache Kafka, a cloud based IoT service, etc.apps
- Applications for use in an Internet of Things environment.analytics
- Analytics for use by Edgent applications.utils
- Optional utilities for Edgent applications.console
- Development console that allows visualization of the streams within an Edgent application during development.samples
- Sample applications, from Hello World to some sensor simulation applications.android
- Code specific to Android.test
- SVTPlaceholder: see EDGENT-23
A couple of key items in the mean time:
rat
build task to check license headersSLF4J is used for logging and tracing.
Search the code for org.slf4j.LoggerFactory to see a sample of its use.
Edgent‘s primary development environment is Java 8, to take advantage of lambda expressions since Edgent’s primary API is a functional one.
However, in order to support Android (and Java 7), other features of Java 8 are not used in the core code. Lambdas are translated into Java 7 compatible classes using retrolambda.
Thus:
In general, most code is expected to work on Android (but might not yet) with the exception:
The Edgent code is in ASF resident git repositories:
https://git-wip-us.apache.org/repos/asf/incubator-edgent.git
The repositories are mirrored on GitHub:
https://github.com/apache/incubator-edgent
Use of the normal GitHub workflow brings benefits to the team including lightweight code reviewing, automatic regression tests, etc. for both committers and non-committers.
For a description of the GitHub workflow, see:
https://guides.github.com/introduction/flow/ https://guides.github.com/activities/hello-world/
In summary:
Since the GitHub incubator-edgent repository is a mirror of the ASF repository, the usual GitHub based merge workflow for committers isn’t supported.
Committers can use one of several ways to ultimately merge the pull request into the repo at the ASF. One way is described here:
Notes with the above PR merge directions:
$ git remote add mirror https://github.com/apache/incubator-edgent.git
The Edgent Git repository contains Eclipse project definitions for the top-level directories that contain code, such as api, runtime, connectors.
The Git repository does not include the 3rd party JARs that Edgent depends on, and Eclipse builds of Edgent projects will fail until a Gradle task is run to make them available in your workspace. See the steps below.
Using the Eclipse Git Team Provider plugin allows you to import these projects into your Eclipse workspace directly from your fork.
git@github.com:<username>/incubator-edgent.git
). The other fields will be populated automatically. Click Next. If required, enter your passphrase.master
) and click Next$ ./gradlew setupExternalJars
The _edgent
project exists to make the top-level artifacts such as build.gradle
manageable via Eclipse. Unfortunately, folders for the other projects (e.g., api
) also show up in the _edgent
folder, and are best ignored.
Builds and JUnit testing of Edgent in Eclipse are independent from the artifacts generated by the Gradle build tooling. Neither environment is affected by the other. This is not ideal, but it's where things are at at this time. Both sets of tooling can be, and typically are, used in the same workspace.
Note: Specifics may change depending on your version of Eclipse or the Eclipse Git Team Provider.