Development and Testing


Building the Kafka feed provider is a simple matter of running a docker build command from the root of the project. I suggest tagging the image with a memorable name, like “kafkafeedprovider”:

docker build -t kafkafeedprovider .


Now we need to start the provider service. This is also a simple matter of running a docker run command, but the details are a little tricky. The service relies on a number of environment variables in order to operate properly. They are outlined below:

Mandatory Environment Variables

DB_URLURLThe base URL for persistent storage (either CouchDB or Cloudant)
DB_USERStringUsername for your DB credentials
DB_PASSStringPassword for your DB credentials

Optional Environment Variables

INSTANCEStringA unique identifier for this service. This is useful to differentiate log messages if you run multiple instances of the service
LOCAL_DEVBooleanIf you are using a locally-deployed OpenWhisk core system, it likely has a self-signed certificate. Set LOCAL_DEV to true to allow firing triggers without checking the certificate validity. Do not use this for production systems!
PAYLOAD_LIMITInteger (default=900000)The maximum payload size, in bytes, allowed during message batching. This value should be less than your OpenWhisk deployment's payload limit.
WORKERStringThe ID of this running instances. Useful when running multiple instances. This should be of the form workerX. e.g. worker0.

With that in mind, starting the feed service might look something like:

docker run -e DB_URL=https://myDbHost -e DB_USER=MyDbUser -e DB_PASS=MySuperSecret -p 80:5000 kafkafeedprovider

This example will start the provider service with the specified DB details. The container provides a number of RESTful endpoints which can be accessed on port 5000 inside the container. To expose this port to the rest of the world -p 80:5000 tells Docker to map port 80 of the host machine into port 5000 inside this new container.

After issuing the docker run command, you can confirm the service started correctly by inspecting the container with a docker logs command.

Install Actions

The provided actions also need to be installed to your OpenWhisk deployment. We have automated this with two different shell scripts, one for Message Hub related actions, and one for generic Kafka related actions. These scripts are and, respectively.

Each script requires a number of arguments which are outlined below:

authKeyThe OpenWhisk auth key to use when installing the actions. Typically this would be the auth key for whisk.system
edgehostThe IP address or hostname of the OpenWhisk core system.
dburlThe full URL (including credentials) of the CouchDB or Cloudant account used by the feed service.
dbprefixA prefix to be prepended to the default DB name (ow_kafka_triggers) that will be created by the provider service.
apihostThe hostname or IP address of the core OpenWhisk system that will be used as the hostname for all trigger URLs. In most cases, this will be the same as edgehost.

An example run might look something like:

./ MyOpenWhiskAuthKey staging_db_prefix

In addition, when running multiple instances, the following argument is required |Name|Description| |---|---| |workers|An array of the IDs of the running instances with each ID of the form workerX. e.g. ["worker0", "worker1"]|

When running multiple instances, an example run might look something like:

./ MyOpenWhiskAuthKey staging_db_prefix "[\"worker0\", \"worker1\"]"


To run the automated test suite, you can issue a Gradle command. There are some tests which talk directly to the provider service over REST, and so these tests must know the IP address and port of the running service. This is done by providing the -Dhealth_url, -Dhost and -Dport arguments to Gradle:

./gradlew :tests:test -Dhealth_url= -Dhost= -Dport=80

The value of the host must be the IP/hostname of the Docker host running the service provider container, and the port must be the exposed port number. Additionally, the OPENWHISK_HOME environment variable must be set to the root of the local OpenWhisk directory. Ex: export OPENWHISK_HOME=<openwhisk_directory>.