Apache S4

Overview

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.

S4 0.5.0 is a complete refactoring of the previous version of S4. It grounds on the same concepts (partitioning inspired by map-reduce, actors-like distribution model), but with the following objectives:

• cleaner and simpler API
• robust configuration through statically defined modules
• cleaner architecture
• robust codebase
• easier to develop S4 apps, to test, and to use the platform

We added the following core features:

• TCP-based communications
• state recovery through a flexible checkpointing mechanism
• inter-cluster/app communications through a pub-sub model
• dynamic application deployment
• toolset for creating nodes, adding

Documentation

For the latest information about S4, please visit our website at:

http://inbubator.apache.org/s4

and our wiki, at:

https://cwiki.apache.org/confluence/display/S4/S4+Wiki

Currently the wiki contains the most relevant and up-to-date documentation.

Source code is available here: https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git

How to build

This only applies if you checkout from the source repository or if you download a released source package.

You need a 6+ JDK.

We use gradle http://gradle.org as the build system.

• From the root directory of the S4 project:

./gradlew install

This will build the packages and install the artefacts in the local maven repository.

• Then, build the tools:

./gradlew s4-tools:installApp

This will build the tools so that you can drive the platform through the “s4” command.

Directory structure

• If you have a source package:

  • root directory contains build and utility scripts, as well as informative files
  • config/ directory contains configuration files for source code formatting
  • doc/ directory contains the javadoc
  • gradle/ directory contains libraries used by the gradle build tool
  • lib/ directory contains some other gradle libraries
  • subproject/ directory contains the plaftorm subprojects: base, comm, core, tools, as well as example (example is not fully ported to 0.5.0)
  • test-apps/ directory contains some examples (some of them very trivial are used in regression tests)

• If you have a binary package:

  • root directory contains the s4 utility script as well as informative files
  • doc/ directory contains the javadoc
  • lib/ directory contains :
    • the platform libraries (prefixed with “s4”)
    • the dependencies