Open-source, web-based user interface for use with DataTorrent, a stream-processing platform for developing real-time, big data applications in Hadoop.
After cloning the Malhar repository from Github:
cd front npm install . make build
This creates a dist folder whose contents should be copied into the static file server root on the DataTorrent Daemon.
Tests are written using the Mocha Framework, the Chai Assertion Library, and the Sinon Library. The suite.js file which is located in front/test/ includes all the tests to be run. The individual test files are located in the same directory of the file they each are respectively testing, and by convention end in .spec.js.
To run the tests in the browser, run make test, then go to http://localhost:3334/test/runner.html to view the test results.
To run tests via command line, run make clitest.
The code in this project is organized in a way that is compatible with browserify, a tool that allows you to write projects in Common.js (node.js) conventions (e.g. require, module, exports). Browserify achieves this by adding a bundling step between coding and viewing in the browser, similar to a compile step for CoffeeScript. The best way to streamline this bundling step during development is with beefy, which spins up a server that automatically bundles the code on request.
One project goal that has not been achieved yet is to set up a complete mock implementation of the DT Gateway API to develop against, so the best way to get started developing/extending the Front is to have a Hadoop cluster with DataTorrent and the DT Gateway installed and running. The steps to set up a development environment are as follows:
make bundle and node server in the front/ folder (in two different shells or as two different background processes).http://node0.yourcluster.com:PORT/static/dev.make bundle command starts the beefy server so that every time this URL is hit, your local code gets rebundled.Because of the size of the project, and specifically one browserify transform called ktbr that precompiles templates, the bundle process can take upwards of 8 seconds. To improve this, you can run the make bundle_no_ktbr command, which will significantly reduce bundle time, but also relies on synchronous ajax requests to fetch template files on demand. The problem here is that fetching these files located on your local machine violates the Same-Origin Policy for web browsers. To circumvent, you may run Chrome or Chromium with web security disabled:
/path/to/chromium-or-chrome/executale/Chromium --disable-web-security
At runtime, the variable DT is available on the global namespace and contains all the base components that the Front relies on, such as classes that model aspects of your datatorrent applications and views that are common throughout the project. To view what is on this module, take a look at the file located at front/js/datatorent/index.js.
The file located at front/js/app/pages.js acts as an index for all registered pages, and contains more information therein about how to set up a page. The actual page definitions are located in front/js/app/lib/pages/ and specify a page name, the widgets that can be instantiated, the default widget configurations (called dashboards), and the page-wide models that will be instantiated. The best way to get familiar with these is to review the code.
Widgets are simply enhanced backbone views. It is recommended (but not necessarily required) to have all widgets extend from DT.widgets.Widget, which maps to the base backbone view located at front/js/datatorrent/WidgetView.js.
Widgets live in /js/App/lib/widgets.
To create a boilerplate for a widget, run
./bin/createWidget [-c] _WIDGET_NAME_
This will append “Widget” to the end of your widget name, so it is not necessary to include it here. The new folder that is created will be located at front/js/app/lib/widgets/_WIDGET_NAME_Widget.
If you would like to help improve this project, please follow the guidelines in the section on contributing in the Malhar repository. Additionally, for linting we use JSHint, so please run your code through this before submission.