Apex Changelog

Version 3.1.1

Improvement

  • [APEX-122] - Fail release build when there are dependencies on snapshot version

Task

  • [APEX-81] - Update netlet dependency to release version

Bug

  • [APEX-89] - Javascript error when launching an app under CDH 5.2 (hadoop 2.5.0)
  • [APEX-101] - Negative Memory reported for Application Master
  • [APEX-121] - Making sure that state is transferred to client for Statefull Codec
  • [APEX-126] - handleIdleTime Called Outside Of The Space Between beginWindow and endWindow
  • [APEX-127] - Tags of overridden setter/getter methods are lost if there are no comments
  • [APEX-149] - In secure mode non-HA setup STRAM web service calls are failing
  • [APEX-170] - StringCodec registered with BeanUtils is throwing null pointer

Version 3.1.0

Improvement

  • [APEX-12] - Fix Base Operator To Not Show Name Property In App Builder

Bug

  • [APEX-35] - Test exceptions due to missing directory in saveMetaInfo
  • [APEX-36] - FSStorageAgent to account for HDFS lease when writing checkpoint files
  • [APEX-37] - Container and operator json line file in StreamingContainerManager should not be appended from previous app attempt
  • [APEX-43] - SchemaSupport: TUPLE_CLASS attribute should use Class2String StringCodec
  • [APEX-56] - Controlled plan modification on operator shutdown

Version 3.0.0

  • Add jersey client jar for app execution
  • Must catch NoClassDefFoundError when processing operator classes in jar, previously catching Throwable was changed to catching Exception
  • Do not catch throwable in DTCli and exit immediately when worker thread throws an error
  • Depend on published netlet version
  • Catch throwable when executing command because launching an app can throw java.lang.VerifyError: bad type on operand stack
  • Removed runOnlyOnce when generating dt-git.properties and generate even when not using release profile
  • Undeploy heartbeat requests are not processes if container is idle
  • Fix potential NPE
  • Comment hide the actually type for string types(URI, URL, Class etc) and add 2 missing wrapper types
  • Fixed typo in webservice url for get-physical-plan command
  • Resolve deleting checkpoint in different thread
  • Removed duplicate code and added unit test for json stream codec
  • APEX-11 #resolve added checkpoint metric
  • Have default connect and read timeouts because the jersey defaults are infinity and that blocks threads and those threads can't be interrupted either
  • Removed invalid app data push annotation
  • Use FileContext instead of FileSystem to write the meta json file
  • Comment added required memory and required vcores in the appinfo
  • Comment filter abstract types from getAssignableClasses call and rename initializable to instantiable
  • Deploy all artifacts by default.
  • Comment fix the bug in trimming the graph
  • HA support for stram webservice filter.
  • Removed dependencies in filter to hadoop classes with private audience as their interface has changed from Hadoop 2.2 to 2.6
  • Related doc updates
  • Comment Prune the external dependencies from type graph and break the type circle
  • Fixing class not found issue due to missing dt-common jar.
  • Resolve removed old alert support in stram
  • Use tokens from Credentials as UserGroupInformation.getTokens() returns HDFS Namenode hidden tokens that should not be passed to containers.
  • Support for RM delegation token renewal in secure HA environments
  • Resolve fixed bug when custom metric keys are not stored correctly in meta file
  • Comment Use apache xbean shaded asm to resolve jdk 8 class and avoid conflict with old asm version that hadoop depends on
  • APEX-5 #resolve Set APPLICATION_PATH for unit tests that launch an app
  • Resolve Added dependency on dt-engine in test scope of archetype
  • Corrected the place to get pom.properties for version info
  • Token authentication support for buffer server
  • Adding default aggregator for primitive customMetrics
  • Netlet has a few fixed related to non firing OP_CONNECT call, so test against those.
  • Print stack trace when exception occurs
  • Commented out app master mb in properties.xml
  • Ability to extract javadocs as xml
  • Moved PAYLOAD_VALUE out of default case. Count number of skipped payload tuples.
  • Resolve fixing ASM signature visitor to add upper bound to Object for unbounded wild card entries
  • Removing unused imports
  • Addressing review comments
  • Reverting code format change
  • Resolve Added END_STREAM to be distributed to physical nodes. Introduced debug message for default case.
  • Comment removed the coding convention
  • Resolve Changed reading of portTypeInfo, port annotations in operator discoverer to read from ASM
  • Added test case for operator discoverer
  • Updated Type Discovery tests to getPortInfo via ASM instead of reflection API
  • Addressing review comments
  • Removed condition for public, final, transient ports when generating typeGraph using ASM
  • Added a class for storing input, output port annotations
  • Added port type info handling for ports other than DefaultInputPort and DefaultOutputPort types
  • Updated a test case for the same
  • Adding annotations to list.. Missed in previous commit
  • Added rescan of typegraph to capture operator ports
  • Changing ASMUtil utility methods to public
  • Addressing review comments.
  • Correcting jar entry name
  • Updated a test after merge
  • Removed system.out print from test cases and added debug print on failure
  • Removing temp file
  • Reversed the order of setting jvm_options for app master
  • Resolve Added validation for root operator should be input operator in logical plan validation
  • Fixing formatting and added lincense header
  • Removed extra line spaces
  • Correcting output port in dt-site.xml stream connection
  • Resolve Added validation for root operator should be input operator in logical plan validation
  • Fixing formatting and added lincense header
  • Removed extra line spaces
  • Correcting output port in dt-site.xml stream connection
  • Removing white spaces
  • Renaming the properties to reflect the code change
  • Comment add the missing open tag
  • Removed unnecessary depedency
  • Resolve Remove unnecessary unpack in app package archetype
  • Resolve support java.lang.Class and resolve uiType to special types
  • Resolve update dependency to dt-common in archetype
  • Resolve added the container jvm options for app master
  • Resolve attach apa file as an artifact
  • Resolve test app package should depend on dt-common instead of dt-engine
  • Removed additional license header
  • Removed incorrectly left in license headers due to incorrect formatting
  • Put back revision info.
  • Added a TODO note for handling of stram delegation tokens for future
  • Comment Add dag attributes to LogicalPlanSerializer
  • Using resolveType method in OperatorDiscoverer to describe an attribute completely
  • Switch to Java7 and update compiler plugin.
  • Remove invalid module references.
  • Separated out HA token creation from non-HA case as it involves special handling that is subject to change if Hadoop's internals change.
  • Resolve - Add all resource managers to RM delegation token service.
  • Fix CLI script.
  • Stram directory is moved to engine
  • Fix depenency plugin version (2.3 shipped with maven 3.2.5 pulls dependencies from test scope)
  • Setup the pom files for OS with optimized dependencies
  • Changed the license header to Apache 2.0 license.

Version 2.1.0

Bug

  • [SPOI-3732] - Get User Info APIs creates users
  • [SPOI-3816] - Ingestion UI: Node server should store pipelines on hdfs rather than local filesystem
  • [SPOI-3820] - Ingestion UI: Rest calls to gateway from Node server fails
  • [SPOI-3845] - BlockSynchronizer sometimes misbehaves when BlockReader is killed.
  • [SPOI-3862] - Ingestion UI - left/right margins have no width
  • [SPOI-3876] - Review the changes done to DimensionsComputation for HadoopWorld Demo
  • [SPOI-3940] - Demo app package version stuck at v1.0-SNAPSHOT
  • [SPOI-4029] - PiJavaScript demo pi calc operator fails to re deploy
  • [SPOI-4067] - Launch of ingestionApp using DTCP is failing
  • [SPOI-4070] - Stray directory /opt/datatorrent/current/datatorrent is created after installation.
  • [SPOI-4077] - Ingestion UI: Create option to scan the directory recursively
  • [SPOI-4079] - Gateway does not retain listen address specified during installation
  • [SPOI-4080] - Gateway guard hides errors with restarts
  • [SPOI-4081] - Ingestion UI: pipeline table doesn't auto refreshes
  • [SPOI-4085] - UI: wrong application selection after sorting in pipeline instances table
  • [SPOI-4087] - Relaunch functionality keeps relaunching multiple apps
  • [SPOI-4094] - Ingestion app tests failing and empty directory getting created under target that are not removed
  • [SPOI-4128] - Need to explain difference between properties & state holding data structure
  • [SPOI-4175] - Remove directory prop from base block reader and ftp block reader from ingestion
  • [SPOI-4229] - committed function not getting invoked in local mode
  • [SPOI-4352] - App Builder uiType not present for several classes
  • [SPOI-4362] - Remove dependencies on DirectoryScanner from OperatorDiscoverer when used in stram
  • [SPOI-4691] - DT console shows negative memory size for killed application
  • [SPOI-4702] - IndexOutOfBoundsException in Stram due to counters in AbstractBlockReader

Improvement

  • [SPOI-3592] - Clean up appInstance widget definitions
  • [SPOI-3783] - Remove gateway port automatic re-selection on startup feature
  • [SPOI-3932] - demos ui references web services v1
  • [SPOI-3944] - Ingestion UI : Display pipelines progress
  • [SPOI-3945] - UI: Add multiple input support
  • [SPOI-3946] - Create an auto scaling scheme for the Block Reader and Writer
  • [SPOI-4003] - Ingestion app: add a feature to re-try failed blocks
  • [SPOI-4026] - Aggregated counters are not published via web-socket in the logical operators topic
  • [SPOI-4048] - Block application launch during critical system issues
  • [SPOI-4078] - Gateway address argument validation during installation and launch
  • [SPOI-4551] - Enhance Gateway API to return AppIDs for a given application name
  • [SPOI-4578] - Allow supporting archive jars in configuration definition for app packages
  • [SPOI-4728] - Support -originalAppId when launching apps through the DT Gateway API
  • [SPOI-4751] - Upgrade sniffer maven plugin to the latest version

New Feature

  • [SPOI-3028] - [HDHT] Hive Interoperability
  • [SPOI-3029] - [HDHT] Export to ORC Format
  • [SPOI-4050] - AppData UI Dashboard page
  • [SPOI-4093] - Ingestion UI: Control for triggering scan of files
  • [SPOI-4636] - Added $? feature in dtcli and use it in CLIProxy
  • [SPOI-4718] - CLI commands to support variable arguments

Task

  • [SPOI-3242] - Rethrow the exceptions being caught in catch block of operators.
  • [SPOI-3502] - Create the landing page for the UI of ingestion application
  • [SPOI-3504] - [Ingestion UI] Create views for creating a pipeline and its description
  • [SPOI-3762] - Provide installation version option for installer
  • [SPOI-3763] - Update document export tokens
  • [SPOI-3870] - Implement transformation functions for rainier poc1
  • [SPOI-3871] - Filter records based field values for rainier poc1
  • [SPOI-3872] - Generate audit and bad record logs for rainier poc1
  • [SPOI-3970] - Ingestion UI : Use the console package
  • [SPOI-3981] - Remove unused imports
  • [SPOI-4040] - Add an operator that keeps track of failed files
  • [SPOI-4122] - [UI] : Display list of skipped files on UI when overwrite flag is false
  • [SPOI-4172] - After effects of removing threshold property from BlockReader in ReaderWriter partitioner/stats-listener
  • [SPOI-4369] - [AppData][AppDataTracker] Deserializer operator
  • [SPOI-4385] - Create metrics aggregators - sum, min, max, count
  • [SPOI-4544] - Custom metrics can be cumulative or per window which can be statically declared by the operator developer
  • [SPOI-4653] - Create Aggregators registry in AppDataTracker
  • [SPOI-3767] - Auto scaling of BlockReader using partitioning
  • [SPOI-4748] - Add data query to custom metrics store

Bug

  • [MLHR-1614] - AbstractFSWriter in append mode is not fault-tolerant
  • [MLHR-1620] - Remove close file from AbstractFSWriter
  • [MLHR-1637] - Cleanup skipping of records from setup of FSDirectoryInputOperator
  • [MLHR-1643] - FileSplitter recovery fails
  • [MLHR-1644] - Add Mock Server Libraries in unit tests of database/key value store operators
  • [MLHR-1653] - Remove JavaScriptOperatorBenchMark form library
  • [MLHR-1656] - AbstractFileOutputOperator LeaseExpired exception when cache reaches its threshold
  • [MLHR-1668] - Stateless Partitioner in case of parallel partition ignores parallel partition count (except the first time define partitions is called)
  • [MLHR-1687] - AbstractBlockReader threshold breaks the idempotency
  • [MLHR-1708] - Duplicate data read from kafka if kafka partitions are less than DT partitions
  • [MLHR-1712] - The directory under which the idempotent state is stored should be relative to the app directory so that the state is copied on relaunch
  • [MLHR-1723] - FTPStringInputOperatorTest fails on Windows OS

Improvement

  • [MLHR-1547] - Integration of Idempotent storage manager to Directory scanner
  • [MLHR-1621] - Parititon Couchbase Output Operator
  • [MLHR-1632] - Add couchbase mock to couchbase tests.
  • [MLHR-1634] - Improve the BlockReader partitioning scheme to accommodate ingestion rate
  • [MLHR-1641] - Improve the BlockReader
  • [MLHR-1661] - Ability to override the stream-codec of input port in AbstractFileOutputOperator
  • [MLHR-1684] - Improve file splitter not to emit all files in one shot and hold scanned file names in memory
  • [MLHR-1694] - BasicCounters improvements

New Feature

  • [MLHR-1355] - Supports secure hadoop cluster in the installer
  • [MLHR-1497] - Operators for ElasticSearch
  • [MLHR-1578] - UI Auth: UI to show different tabs and/or buttons for different user permissions

Version 2.0.1

Bug

  • [SPOI-4379] - Operator removed from physical plan due to invalid SHUTDOWN status
  • [SPOI-4381] - Queue size missing on physical operator page
  • [SPOI-4382] - queueSize port metric reports bogus values
  • [SPOI-4384] - Recovery fails due to corrupted checkpoints.
  • [SPOI-4437] - DTCli not recognizing $HOME variable set

Improvement

  • [SPOI-4728] - Support -originalAppId when launching apps through the DT Gateway API
  • [SPOI-4390] - Replace ‘start time’ on console to ‘Up time’

Version 2.0.0

Bug

  • [SPOI-4046] - Gateway /containers?states={state} call returns erroneous state information
  • [SPOI-4057] - License can not be upgraded to Evaluation / Production License
  • [SPOI-4037] - CONTAINERS_MAX_COUNT in sandbox prevents demos launch from dtcli
  • [SPOI-4008] - HDHT Error while flushing write cache
  • [SPOI-3837] - Partitioner interface semantics broken in 2.0.0 physical plan implementation]
  • [SPOI-3901] - twitter demo app package launch throws NoClassDefFoundError
  • [SPOI-3900] - Gateway gets 400 error when phone home
  • [SPOI-3922] - dtcli does not show output port attributes.
  • [SPOI-3934] - DT CLI Needs to validate user inputs
  • [SPOI-3054] - # Of Tuples Produced != Number Of Tuples Output By Unifier When Repartitioning
  • [SPOI-3210] - HDHT DTFile reader bug
  • [SPOI-3340] - Settings in ~/.dt/dt-site.xml don't override app package defaults
  • [SPOI-3349] - Can we document how to enable password security in gateway
  • [SPOI-3365] - Launch App Package does not honor -local option
  • [SPOI-3369] - Implement the pam authentication as a hadoop authentication handler
  • [SPOI-3396] - launch command should be able to specify local conf file when launching app package
  • [SPOI-3397] - DT Gateway doesn't start on Mapr 4.0.1
  • [SPOI-3402] - Checking the UID for Dtadmin
  • [SPOI-3419] - KafkaAdsDimensionsDemo demo available with installer fails
  • [SPOI-3420] - Installation issues on HDP 2.2
  • [SPOI-3469] - gateway, dtcli fail to start in development mode
  • [SPOI-3503] - AppPackage tests produce 100+GB files
  • [SPOI-3522] - MaxEventsPerSecond for Flume Ingestor should adjust for partitioned instances
  • [SPOI-3528] - Installer does not show relevant error if nonexistent env file passed as argument and finishes silently with defaults
  • [SPOI-3548] - browser inconsistencies with table
  • [SPOI-3550] - Confusing Breadcrumb navigation on Application Page
  • [SPOI-3555] - Unifier names should not be links to nonexistent pages
  • [SPOI-3594] - Systems Diagnostic produces conflicting results for Hadoop installation
  • [SPOI-3595] - Correctly Implement the clone method in platform.
  • [SPOI-3608] - User Profile and User Management should not show when auth is disabled
  • [SPOI-3613] - RBAC Configuration screen is completely missing after recent changes
  • [SPOI-3628] - Pubsub websocket auth not working with kerberos frontend authentication
  • [SPOI-3638] - Cant assign roles to user in console
  • [SPOI-3644] - Develop Tab Can not be seen on naviagation bar
  • [SPOI-3646] - Error creating new config xml file in package
  • [SPOI-3647] - MR Operator demo throws null pointer exception
  • [SPOI-3648] - Cant add admin permissions to existing roles
  • [SPOI-3654] - In passwd policy user can not change own password using ui console
  • [SPOI-3657] - Can't Launch Apps Using ui console if user is not the same as dtgateway user
  • [SPOI-3663] - modify certification scripts and gateway proxy to upload demo jars from repackaged malhar demos
  • [SPOI-3665] - DataTorrent logo link broken in console
  • [SPOI-3670] - Update DTCliTest to create app package like AppPackageTest.java
  • [SPOI-3672] - Uploaded license not used when launching app
  • [SPOI-3674] - License info not updated after license upload until page reload
  • [SPOI-3675] - Display Error in Sandbox. License Manager Error:null
  • [SPOI-3682] - doubleclick select active kills containers
  • [SPOI-3684] - Gateway deletes JSON app silently (no error) on PUT
  • [SPOI-3685] - Error message in launch app modal is not red
  • [SPOI-3686] - Launch properties do not get sent with the launch request
  • [SPOI-3689] - AppPackages APIs give 404 not found messages
  • [SPOI-3690] - APIs falsely report role assignments
  • [SPOI-3691] - API /ws/v1/config/hadoopInstallDirectory does not work
  • [SPOI-3708] - Gateway shows License error after license agent gets back to normal
  • [SPOI-3710] - Sandbox packaging invalid due to HDFS dependency
  • [SPOI-3712] - Exclamation point in DT cli tries to expand “event”
  • [SPOI-3713] - When CLI has a problem running a CLIProxy command, there is out of memory error from the gateway
  • [SPOI-3714] - Launch button from other screens should bring up launch modal
  • [SPOI-3716] - locality performance benchmarks operators failing with OOM exception
  • [SPOI-3717] - Insufficient license memory for Node Local Performance benckmark application
  • [SPOI-3718] - change high availability certification benchmarking to compare number of active containers instead of total number of containers
  • [SPOI-3720] - License manager not found returned from Gateway even though it returns valid license including agent info
  • [SPOI-3721] - Changes to XMLConfig to incorporate changed properties
  • [SPOI-3725] - NullPointerException in Certification-initDemo for PerformanceBenchmarkForFixedNumberOfTuples
  • [SPOI-3729] - Thread local validation failure for multiple ports between two operators
  • [SPOI-3732] - Get User Info APIs creates users
  • [SPOI-3733] - Can't Launch Apps Using UI or gateway APIs in kerberos environment
  • [SPOI-3734] - Remove macro from cli for application or add relevant application
  • [SPOI-3754] - Gateway can only restore roles once

Improvement

  • [SPOI-2036] - Ensure that definePartitions and partitioned are called for parallel partitioned operators
  • [SPOI-2955] - Provide tool-tip capability to RTS UI
  • [SPOI-3194] - Support PAM authentication mechanism as an agent for LDAP
  • [SPOI-3304] - Reorganize code to ensure easy consumability
  • [SPOI-3345] - Delete OperatorContext.PartitionTPSMax and .PartitionTPSMin
  • [SPOI-3346] - Delete OperatorContext.InitialPartitionCount
  • [SPOI-3366] - Update Eval license for RTS 2.0
  • [SPOI-3401] - Switch installer, gateway, console from using HADOOP_PREFIX to hadoop executable
  • [SPOI-3404] - Provide support for viewing/collecting YARN logs across the cluster
  • [SPOI-3415] - Remove defaults from dt-site.xml and add version
  • [SPOI-3418] - Remove dependency on /etc/datatorrent
  • [SPOI-3447] - Partitioning of couchbase input operator
  • [SPOI-3525] - Messages shown in installer and uninstaller are longer than 80 characters
  • [SPOI-3554] - Log-Level Setter for angular
  • [SPOI-3626] - appPackageDisplayName in /ws/v1/appPackages
  • [SPOI-3667] - Confirmation box to shutdown or kill apps should have “warm” colors
  • [SPOI-3668] - DTGateway logs are flooded with permission WARN logs
  • [SPOI-3694] - Populate launch properties with required properties of app
  • [SPOI-3697] - Add content-disposition so that app package download has a reasonable file name when downloaded from browser
  • [SPOI-3698] - Support URI codec out the box so that UI can set URI operator properties
  • [SPOI-4066] - Portable environment settings across releases

New Feature

  • [SPOI-3393] - Track recordings by a generated ID rather than startTime
  • [SPOI-3422] - Restore default roles with backend
  • [SPOI-3683] - Add download button to app package
  • [SPOI-3753] - “Reset to Defaults” for user roles

Task

  • [SPOI-3927] - Support per application configuration defaults and required properties
  • [SPOI-2809] - Implement password authentication as a hadoop authentication type
  • [SPOI-3115] - Update document with password authentication once UI supports user mgmt
  • [SPOI-3239] - HDHT store removal support
  • [SPOI-3351] - Abstract the reconciler used in Guava and Rainier to malhar
  • [SPOI-3368] - Configuration handling on major version upgrade
  • [SPOI-3371] - CLI to doAs the user specified by HADOOP_USER_NAME if security is enabled
  • [SPOI-3372] - Gateway REST service to only return rows the user is authorized to view
  • [SPOI-3373] - Gateway Websocket service to only publish to subscribers with info individual subscribers are authorized to view
  • [SPOI-3374] - Gateway to return 403 Forbidden for resources the user is not authorized to view or operations the user is not authorized to perform
  • [SPOI-3375] - Gateway support for permission based authorization
  • [SPOI-3376] - Gateway REST call to show what permissions the current user has
  • [SPOI-3377] - Gateway REST call to support role and permission management
  • [SPOI-3378] - HTTP Header auth support in Gateway
  • [SPOI-3381] - Implement Kerberos group to role mapping
  • [SPOI-3385] - Gateway ACL support at the application level
  • [SPOI-3390] - LDAP/AD support in Gateway
  • [SPOI-3405] - Implement Kerberos security context and security context filter in Gateway
  • [SPOI-3443] - Ingestion Application repo and package
  • [SPOI-3472] - Installer testing
  • [SPOI-3485] - Console repository relocation
  • [SPOI-3486] - Create HDHT section in Application Developer Guide
  • [SPOI-3517] - Document JVM_OPTIONS and QUEUE_NAME features in user docs
  • [SPOI-3519] - Add PAM auth user docs
  • [SPOI-3520] - Update RBAC user docs
  • [SPOI-3521] - Console user docs updates
  • [SPOI-3536] - Create hdfs output operator to write table files
  • [SPOI-3537] - write hdfs writer for small files to copy to vertica
  • [SPOI-3538] - Combine small file writer with vertica writer
  • [SPOI-3540] - Sandbox updates to support app package launches from console
  • [SPOI-3593] - Clean up import functionality for demo apps
  • [SPOI-3610] - Implement RBAC features for App Packages
  • [SPOI-3619] - Update console for the new app package REST API
  • [SPOI-3627] - Convert our REST API from v1 to v2 as we have made some backward incompatible changes in the REST API
  • [SPOI-3631] - Test HDHT recovery
  • [SPOI-3632] - Upgrade and test demos on 2.0.0
  • [SPOI-3633] - Set certain sensitive permissions “admin” role only
  • [SPOI-3634] - Allow default app instance and app package permissions for each user
  • [SPOI-3635] - Remove INITITAL_PARTITION_COUNT from demo app pkg properties
  • [SPOI-3636] - Upgrade and test demos on 2.0.0
  • [SPOI-3650] - Test demos on sandbox on dt 2.0.0
  • [SPOI-3651] - benchmarks for dt 2.0.0
  • [SPOI-3652] - list of current benchmarks
  • [SPOI-3653] - Document property changes needed run demos on 2.0.0
  • [SPOI-3661] - Update security section in operations guide
  • [SPOI-3662] - Operations Guide: Update application configuration section
  • [SPOI-3669] - Remove gateway jars api usage from benchmarking and use app pkg api instead
  • [SPOI-3695] - Documentation on System Alerts in the Gateway
  • [SPOI-3700] - run app memory usage benchmarks for 2.0
  • [SPOI-3701] - run performance benchmarks for 2.0
  • [SPOI-3702] - run high availability benchmarks for 2.0
  • [SPOI-3703] - Run Fs output operator benchmark
  • [SPOI-3704] - run performance across tuple size benchmarks for 2.0
  • [SPOI-3705] - Change jars api usage to apps package api usage in certification
  • [SPOI-3706] - Convert Malhar benchmark module to app package
  • [SPOI-3730] - Getting Started Guide updates

Bug

  • [MLHR-1237] - Scrolling log viewer with mouse does not trigger getting more log content
  • [MLHR-1242] - “last heartbeat” in killed container list shows date in 1969
  • [MLHR-1443] - Attempt reconnect if web socket connection is dropped
  • [MLHR-1523] - Physical operator ids sorted lexicographically
  • [MLHR-1552] - Port JMS Input operator changes to library.
  • [MLHR-1553] - shutdown and kill cmds still visible when ended apps are selected
  • [MLHR-1588] - New installation does not walk through welcome steps
  • [MLHR-1590] - App Instance dashboard cannot find Logical DAG widget for starting app
  • [MLHR-1591] - pending undeploy does not have an icon or color associated
  • [MLHR-1592] - webSocket does not reconnect if user logs out then logs back in
  • [MLHR-1593] - Installer should be ok with 404 errors when fetching HadoopLocation and dfsRootDirectory
  • [MLHR-1594] - DFS permissions error during install does not provide resolution steps
  • [MLHR-1595] - Installer asks for login even if auth is disabled
  • [MLHR-1606] - Change EDIT_AND_KILL_OTHERS_APPS to MANAGE_OTHERS_APPS
  • [MLHR-1609] - Physical Operators have an invalid heartbeat when PENDING_DEPLOY
  • [MLHR-1611] - Do not try to put or edit admin role in auth management page
  • [MLHR-1612] - New containers from websocket do not get jvm name
  • [MLHR-1615] - Feedback for retrieving ended apps
  • [MLHR-1619] - When launching app package, allow launch time parameters to be specified
  • [MLHR-1622] - YahooFinanceApplication throws null pointer exception on 2.0
  • [MLHR-1627] - Make twitter credentials into requiredProperties in app package
  • [MLHR-1631] - NullPointerException in launching HDFSBenchmarking App:TupleSize property missing in appResponse

Improvement

  • [MLHR-1190] - Infinite scroll mechanism in stram events widget
  • [MLHR-1234] - Add version information to system diagnostics screen
  • [MLHR-1555] - Remove dashboard component from main operations page
  • [MLHR-1571] - Height of app list should be determined by available space
  • [MLHR-1596] - Extract two-way infinite scroll behavior into directive
  • [MLHR-1597] - Fix FTP Input operator
  • [MLHR-1602] - Partitioning of couchbase input operator

New Feature

  • [MLHR-1228] - create profile storage object
  • [MLHR-1261] - AngularJS Migration - Tuple Viewer
  • [MLHR-1262] - AngularJS Migration - Tuple Recording
  • [MLHR-1354] - Auth management support in the UI

Story

  • [MLHR-975] - Migration from Backbone to Angular

Task

  • [MLHR-1292] - Recording View
  • [MLHR-1331] - Update the tuple recording topic
  • [MLHR-1586] - Remove DirectoryScanInputOperator from library
  • [MLHR-1601] - Re-arrange logstream app in dt application package format
  • [MLHR-1608] - Create a Block reader which emits Slice and doesn't read ahead of a block boundary
  • [MLHR-1617] - Sensitive permission assignment need dialog box warning with consequences
  • [MLHR-1618] - Update console for the new app package REST API
  • [MLHR-1623] - Create synchronizer for asynchronously processing streaming data for committed windows
  • [MLHR-1626] - Write unit tests for AbstractSynchronizer

Bug

  • [SPOI-2946] - Provide a tool that generates a license report based on license audit logs
  • [SPOI-2974] - Better error handling when license is expired
  • [SPOI-3147] - Issues with dynamic partitioning of Block readers
  • [SPOI-3192] - Send proxy user when making web service calls from gateway
  • [SPOI-3267] - HDHT Operator crash when launching application against existing store
  • [SPOI-3271] - Change port stats propogation test to check for cumulative buffer server bytes
  • [SPOI-3286] - HDHT WAL recovery crashing with NegativeArraySizeException
  • [SPOI-3287] - Research the types of data that can be stored in hive orc files
  • [SPOI-3288] - NullPointerException in StreamingContainerManager-> fillLogicalOperatorInfo
  • [SPOI-3289] - App Packages launch not reading required properties from dt-site.xml
  • [SPOI-3293] - Incorrect application name when launching JSON app.
  • [SPOI-3294] - Configuration setting from property.xml not effective for JSON app
  • [SPOI-3296] - npm install fails for malhar-ui-console
  • [SPOI-3297] - Remove keys from metric list in app dashboard widget.
  • [SPOI-3300] - Two operators named “Dimension Computation”
  • [SPOI-3305] - Create Sales Demo JSON Input Generator
  • [SPOI-3306] - Change Event Schema to default to Sales Schema
  • [SPOI-3309] - DT flume sink not draining under some circumstances
  • [SPOI-3313] - Setting attributes with JSON and properties app needs cleanup
  • [SPOI-3319] - NPE in FSStatsRecorder
  • [SPOI-3320] - The platform should use user folder in hdfs for storage
  • [SPOI-3323] - Gateway resource leak when RM is not running and/or during network problem
  • [SPOI-3325] - HDHT DTFile clean cache when reader close
  • [SPOI-3338] - Application level operator properties do not override “global” operator properties
  • [SPOI-3343] - Gateway was stuck

Improvement

  • [SPOI-2918] - Document Counters
  • [SPOI-3188] - Ability to interpret an input operator as a regular operator
  • [SPOI-3312] - OperatorAnnotation that enforces checkpointing to happen at application window boundary
  • [SPOI-3317] - Add sales generator tuples per window randomization controls
  • [SPOI-3318] - Add more dimensions and aggregates to Sales demo
  • [SPOI-3321] - Improve data variation between categories, regions, and discounts
  • [SPOI-3329] - Implement simple variable substitution in properties files

New Feature

  • [SPOI-2014] - Support for doc link for an operator
  • [SPOI-2931] - Make sure app packages work in sandbox
  • [SPOI-2989] - HDHT - Recovery

Task

  • [SPOI-2663] - Certify datatorrent on a CDH5 secure cluster
  • [SPOI-2702] - HDS - Generic time series query
  • [SPOI-2843] - Tracker ticket for work related to app package in Malhar
  • [SPOI-2945] - Allow application to start even if license manager is not running for production license file
  • [SPOI-2950] - Generate monthly alerts when the memory used crosses the licensed memory
  • [SPOI-3022] - Run jpa and jdbc tests on vertica installation in cluster
  • [SPOI-3098] - Scrub all components that make it to the release list of jiras
  • [SPOI-3119] - Generate audit logs in the applications as well
  • [SPOI-3153] - Certify datatorrent on HDP secure cluster
  • [SPOI-3156] - Tool to generate license report from a single Hadoop grid
  • [SPOI-3158] - Webservice for license report
  • [SPOI-3160] - Uptime debugging for license mgr
  • [SPOI-3161] - license mgr should write memory reported by each app in the logs
  • [SPOI-3184] - Test case to reproduce locality issue in CDH
  • [SPOI-3229] - New app will launch for production license even if no memory is available in license
  • [SPOI-3261] - Kafka one to many dynamic partitioning pilot
  • [SPOI-3269] - Support contact update
  • [SPOI-3278] - Research elastic search functionality to build operators
  • [SPOI-3280] - Create a model parser that loads the model mapping file and generates sql mapping
  • [SPOI-3282] - Create a vertica output operator
  • [SPOI-3283] - Start License Manager as part of Installation
  • [SPOI-3284] - Watch dog process for License Manager in Gateway
  • [SPOI-3299] - O15 demo: Dictionary for keys
  • [SPOI-3307] - Add Enrichment operator data file
  • [SPOI-3315] - O15 demo: Update widget configuration for sales schema
  • [SPOI-3335] - Kafka offset manager
  • [SPOI-3339] - Move FSStorageAgent to Malhar
  • [SPOI-3352] - Process OperatorCommand returned by stats listener
  • [SPOI-3411] - Create a feed processor operator that converts input event to table rows
  • [SPOI-3412] - Create gzip input operator
  • [SPOI-3413] - Create pluggable functions for each column based given function name and parameters
  • [SPOI-3414] - Create Table controller to handle value generation for each column including user defined functions
  • [SPOI-3561] - Recording View: links to start and stop recording
  • [SPOI-3562] - Recording View: page module
  • [SPOI-3565] - Container Page (angular)

Bug

  • [MLHR-1267] - AngularJS Migration - Dashboard Layout Lock
  • [MLHR-1352] - Custom directive for breadcrumbs
  • [MLHR-1489] - Create a hive output operator that can write to hive orc files
  • [MLHR-1500] - Cleanup of the junit.framework.Assert
  • [MLHR-1505] - Stram event error closes when clicking on stack trace
  • [MLHR-1506] - Stram event collection removes events
  • [MLHR-1512] - UI Console - Dashboard Reset
  • [MLHR-1532] - Table says “loading” when active filter result is 0 rows
  • [MLHR-1541] - Add orc file output to adsdimension demo
  • [MLHR-1542] - App Data Framework - WebSocket Support
  • [MLHR-1585] - Installer fails with auth enabled

Improvement

  • [MLHR-1256] - Create a new landing dashboard
  • [MLHR-1432] - Add confirm for deleting an app package
  • [MLHR-1451] - Loading feedback for malhar-angular-table
  • [MLHR-1452] - Loading feedback for app packages and package apps
  • [MLHR-1454] - Performance Tuning for malhar-angular-table
  • [MLHR-1499] - Idempotent State Manager
  • [MLHR-1531] - App overview does not show started time
  • [MLHR-1540] - Remove stram events from physical view
  • [MLHR-1543] - Switch to gulp in malhar-angular-table
  • [MLHR-1549] - Short operator properties should be shown in line

New Feature

  • [MLHR-1282] - AngularJS Migration - App Instance Page - Alerts Recordings View
  • [MLHR-1301] - AngularJS Migration - Physical Operator Page - Recordings
  • [MLHR-1317] - AngularJS Migration - App Instance Page - Logical View - Log Levels
  • [MLHR-1362] - AngularJS Migration - App Instance Page - Stram Events Widget Resize
  • [MLHR-1370] - AngularJS Migration - Container View - Chart
  • [MLHR-1428] - Dashboard Component - Option to Disable Vertical Resize
  • [MLHR-1462] - Socket.IO Kafka Communication Protocol
  • [MLHR-1463] - UI Console - Client Settings
  • [MLHR-1464] - Kafka App Data - Server Socket.IO Node.js
  • [MLHR-1465] - Kafka Socket.IO Service - Latency
  • [MLHR-1466] - Kafka Server - Query Cache
  • [MLHR-1470] - Dashboard Builder Integration
  • [MLHR-1486] - UI Console - Deploy Scripts
  • [MLHR-1488] - App Package Dag Viewer - Handle Case when DAG is empty
  • [MLHR-1491] - App Data UI - Runtime Configuration for Default Dashboard/Widgets/Queries
  • [MLHR-1495] - Kafka Debug Widget - Kafka Producer/Consumer Topics
  • [MLHR-1498] - Provide Ability to Reset Dashboard Configuration
  • [MLHR-1502] - App Data Server - Fetch Latest Kafka Offset
  • [MLHR-1514] - App Data Server - EventEmitter
  • [MLHR-1515] - App Data Server - UML Diagrams
  • [MLHR-1520] - App Data UI - Table Widget
  • [MLHR-1521] - App Data UI - WebSocket Support
  • [MLHR-1530] - App Data UI - Table Widget

Story

  • [MLHR-1295] - AngularJS Migration - Logical Operator Page
  • [MLHR-1296] - AngularJS Migration - Physical Operator Page

Task

  • [MLHR-1270] - Container Page (angular)
  • [MLHR-1272] - Container Page: memory gauge
  • [MLHR-1274] - Container Page: metric chart
  • [MLHR-1281] - Stream View
  • [MLHR-1286] - Stream View: sources table
  • [MLHR-1287] - Stream View: sinks
  • [MLHR-1288] - Port View
  • [MLHR-1290] - Port View: overview
  • [MLHR-1291] - Port View: chart
  • [MLHR-1293] - Recording View: page module
  • [MLHR-1294] - Recording View: links to start and stop recording
  • [MLHR-1307] - Installation Wizard
  • [MLHR-1310] - Info Menu Links
  • [MLHR-1312] - Lock Layout (malhar-angular-dashboard)
  • [MLHR-1453] - System Diagnostics Page
  • [MLHR-1460] - Installation Wizard: welcome page
  • [MLHR-1461] - Installation Wizard: hadoop config page
  • [MLHR-1467] - Installation Wizard: license screen
  • [MLHR-1482] - Installation Wizard: summary screen
  • [MLHR-1483] - Installation Wizard: license upload
  • [MLHR-1496] - Create an operator for Solr
  • [MLHR-1516] - Annotate new FS output operators
  • [MLHR-1518] - WebSocket support for dimension demo
  • [MLHR-1574] - UI Auth: support for current password auth
  • [MLHR-1575] - UI Auth: Managing role and permissions

Sub-task

  • [SPOI-3123] - Add troubleshooting section to Operations and Installation Guide
  • [SPOI-3124] - Check for vmem/pmem ratio during System Diagnostics test
  • [SPOI-3129] - Integrate Kafka query protocol for AdsDimension demo.
  • [SPOI-3154] - Setup AdsDimension demo on cluster
  • [SPOI-3226] - Create JSON AdInfo Generator for App Builder Demo
  • [SPOI-3227] - Crate JSON to Map converter for App Builder Demo
  • [SPOI-3228] - Create shared dimensions computation schema for App Builder Demo

Bug

  • [SPOI-1667] - Drop of Performance for Machine demo
  • [SPOI-2742] - Core dumps on EMR
  • [SPOI-2942] - Disable physical plan locking when there is no available license memory
  • [SPOI-2947] - Provide a tool that distributes memory and generates licenses that can be deployed on multiple clusters
  • [SPOI-3018] - Investigate raw performance of vertica jdbc
  • [SPOI-3024] - Adding Virtual Mem to Physical Memory setting in Readme file
  • [SPOI-3025] - Can't see the container logs from DT Console in Pivotal installation
  • [SPOI-3053] - Continual Repartitioning Of Operator Causes Out Of Memory Exception
  • [SPOI-3081] - Core StatsTest.testPortStatsPropagation Failing
  • [SPOI-3107] - Failed to load Logical Plan
  • [SPOI-3132] - User Interface Guide URL specified in current Sandbox gives 404
  • [SPOI-3181] - dtgateway script: $PATH is being appended to repeatedly when gateway has trouble starting
  • [SPOI-3187] - HDHT query results partition duplication
  • [SPOI-3190] - HDHT AdsDimension demo last bar should grow
  • [SPOI-3195] - Operator class exception when loading app package
  • [SPOI-3196] - CLI returns Perm Gen Space errors when there are too many classes to inspect in app package
  • [SPOI-3197] - Provide meta information for operators that can be used by DAG builder
  • [SPOI-3198] - App Package archetype is erroneously including app package manifest in the low level jar file
  • [SPOI-3213] - Newer maven versions (>3.1) do not work with copy-maven-plugin in app package archetype
  • [SPOI-3215] - Duplicate streams in OperatorDeploymentInfo with default partition/parallel partitioning
  • [SPOI-3232] - Mobile demo not dynamically partitioning
  • [SPOI-3262] - Issue with serialization of xml cartesian product operator
  • [SPOI-3319] - NPE in FSStatsRecorder

Improvement

  • [SPOI-1967] - Accept Password for Private Key
  • [SPOI-2997] - License file content updates
  • [SPOI-3083] - Ability to save DAG as a DT App Package
  • [SPOI-3084] - Operator Library Browser
  • [SPOI-3087] - Properties Section
  • [SPOI-3088] - Attributes Section
  • [SPOI-3089] - Operators needed for demo
  • [SPOI-3102] - FileSplitter needs to be idempotent
  • [SPOI-3136] - Add a check to see if stram is connected to THIS gateway

New Feature

  • [SPOI-3142] - Ability to take Logical Plan Serialization JSON to construct a DAG and launch it
  • [SPOI-3148] - App Package REST API needs to allow PUT of json logical plan for adding/replacing application in app package
  • [SPOI-3149] - Ability to launch a json-specified app within app package
  • [SPOI-3185] - Import demo app packages from dtgateway fs
  • [SPOI-3186] - Group app packages under demos folder in release
  • [SPOI-3189] - Add display name and description to app package's manifest
  • [SPOI-3199] - XML Javadoc
  • [SPOI-3266] - Provide Ability to Allow Cross Origin Access

Task

  • [SPOI-3039] - Print data upon license mgr app start
  • [SPOI-3061] - Add port tuple type in logical plan serialization
  • [SPOI-3067] - Remove time-bombed public/private key pair from the code
  • [SPOI-3068] - Make sure the release process (sandbox, installer binary) is okay with the time bound key pair removed from the code
  • [SPOI-3069] - Remove the entire request blob process and generate license straight from customer info and license info
  • [SPOI-3070] - Device ID change in license
  • [SPOI-3071] - Generate License web tool to prompt for password for the private key
  • [SPOI-3126] - O15 - Generic Dimension Computation/Store Operator
  • [SPOI-3128] - HDHT - AdsDimension app integration for internal demo
  • [SPOI-3130] - HDHT - Test files with varying block sizes
  • [SPOI-3134] - Add a organization field in license file
  • [SPOI-3162] - Kafka ingestion demo
  • [SPOI-3164] - Integrate Kafka Query frontend into demos
  • [SPOI-3165] - Ability to return JSON app as-is from app package through REST
  • [SPOI-3166] - Ability to delete JSON apps from app package
  • [SPOI-3167] - Ability to save incomplete json-based app to an app package
  • [SPOI-3179] - Investigate how IDEs parse javadoc
  • [SPOI-3182] - Complete dag annotations spec
  • [SPOI-3193] - Investigate pluggable authorization mechanisms supported by Hadoop
  • [SPOI-3200] - Generate resource file containing javadoc comments and custom tags
  • [SPOI-3201] - Configure default javadoc doclet in pom to include custom tags
  • [SPOI-3202] - Transform xml javadoc resources file to contain only comments and tags
  • [SPOI-3203] - Add javadoc resource file to the class jar at maven build
  • [SPOI-3206] - Add field and method comments to transformed xml javadoc
  • [SPOI-3217] - Limit events DTFlumeSink pumps into the dag
  • [SPOI-3219] - Help diagnose launch issue
  • [SPOI-3225] - Check and fix machine and twitter hash tag demos
  • [SPOI-3236] - displayName should be available from appPackage/applications call

Bug

  • [MLHR-1347] - Create a block reader in library which is capable of dynamic partitioning itself
  • [MLHR-1366] - HdfsBucketStore is not completely covered by unit test cases
  • [MLHR-1367] - Add Counter Aggregators For Monitoring AbstractThroughputFSDirectoryInputOperator
  • [MLHR-1373] - Container log download link
  • [MLHR-1404] - App Package File Upload
  • [MLHR-1407] - App List Kill App Selection Issue
  • [MLHR-1410] - Add a “remove” option to the “set/subscribe” method of BaseCollection
  • [MLHR-1412] - App Package archetype is erroneously including app package manifest in the low level jar file
  • [MLHR-1419] - Xml cartesian product operator has kryo serialization errors in some cases
  • [MLHR-1422] - UI Physical tab-> sorting by operator id is by lexical order not numerical
  • [MLHR-1423] - app instance page does not update when state goes ACCEPTED => RUNNING
  • [MLHR-1426] - favicon not being handled in gulp build
  • [MLHR-1436] - Uptime shows -1 day even when the app is runing for some time
  • [MLHR-1438] - AdsDimensionsWithHDSDemo - Expose Aggregation in Operator Properties
  • [MLHR-1442] - Move the recipient property for the machine data to config
  • [MLHR-1557] - subscribe handler on app instance alters original ws message
  • [MLHR-1558] - Palette in the Apps List does not update after one app gets killed

Improvement

  • [MLHR-1427] - Make app name link to instance page as well in AppsList

New Feature

  • [MLHR-1388] - Base DAG Viewer
  • [MLHR-1389] - App Package List Page
  • [MLHR-1390] - Applications/Package Info Page
  • [MLHR-1391] - App Package REST API Integration
  • [MLHR-1392] - App Package CRUD
  • [MLHR-1406] - UI Support of app package import
  • [MLHR-1408] - Allow Shutting Down/Killing Multiple Applications
  • [MLHR-1424] - UI Grid 3 Integration - Fonts
  • [MLHR-1429] - Dashboard Component - Configurable Widget
  • [MLHR-1430] - Kafka UI Demo
  • [MLHR-1431] - Visual Data Demo Update

Story

  • [MLHR-1241] - App Package support in UI
  • [MLHR-1357] - UI Dashboard Front Page Design

Task

  • [MLHR-1079] - Normalize timestamps for info pages
  • [MLHR-1104] - Design App Package and Upload/Launch Feature
  • [MLHR-1105] - Polish simplified (current) app data alert feature
  • [MLHR-1207] - Convert demo apps to app packages
  • [MLHR-1309] - License Info
  • [MLHR-1363] - Create file splitter that breaks file into blocks and emits block metadata and a block reader
  • [MLHR-1415] - Clean-up Hdfs Output operators in library and incorporate the features of fault-tolerant Writer
  • [MLHR-1503] - Remove deprecated jdbc package from contrib

Version 1.0.4

Bug

  • [SPOI-2346] - ExactlyOnceTest#testLinearInputOperatorRecovery hangs
  • [SPOI-2511] - ResourceManager HA support
  • [SPOI-2625] - uninstall.sh should print message that reminds users of running DT applications
  • [SPOI-2922] - Counters Aggregator gets lost when an operator is parallel partitioned - test fix
  • [SPOI-2939] - AppBundles test resource refers to fixed version
  • [SPOI-2967] - Mobile demo dies after couple of days
  • [SPOI-2979] - App Bundle unit test should load properties.xml
  • [SPOI-2991] - [kafka-yarn] Archive resource to use less space in HDFS
  • [SPOI-2993] - appBundle with conflicting names in config and ApplicationAnnotation can't launch in CLI with either name
  • [SPOI-2995] - Generate MANIFEST.MF automatically instead of requiring user to change it
  • [SPOI-2999] - Investigate application integration to Vertica
  • [SPOI-3012] - launch application throws NPE when yarn.application.classpath is not defined
  • [SPOI-3015] - DTGateway log grows indefinitely
  • [SPOI-3019] - Investigate how many objects per second (with a single field) using JPA can be written to vertica
  • [SPOI-3027] - Sandbox GATEWAY_CONNECT_ADDRESS change to support VMWare
  • [SPOI-3055] - NullPointerException When Repartitioning Too Frequently
  • [SPOI-3074] - NullPointerException in AppMaster
  • [SPOI-3079] - NPE while launching application
  • [SPOI-3080] - Content-disposition: attachment for container logs
  • [SPOI-3091] - Null Pointer Exception/Internal Server Error When Getting Operator Stats
  • [SPOI-3092] - DefaultUnifier of a port in an operator does not function correctly when the port has more than one sinks

Improvement

  • [SPOI-2821] - Add bucket processed time
  • [SPOI-2891] - DTGateway default log setting too verbose
  • [SPOI-2938] - Document AppBundles development and deployment
  • [SPOI-3044] - gateway should show the full stack trace of the origin upon error when proxying

New Feature

  • [SPOI-2720] - API Call for determining if an application's stram can connect to the gateway
  • [SPOI-2981] - HDS - File Format
  • [SPOI-2982] - HDS - Writing Data Files
  • [SPOI-2983] - HDS - Bucket Meta Data
  • [SPOI-2984] - HDS - WAL
  • [SPOI-2986] - HDS - Bucket Management
  • [SPOI-3031] - Add ability to specify number of recording windows
  • [SPOI-3032] - Delete request for recording
  • [SPOI-3033] - Provide byte offset for each line when grepping container log
  • [SPOI-3049] - Ingestion streamlet design for first cut

Task

  • [SPOI-1734] - Licensing Agent Logging for Audit
  • [SPOI-2810] - Update document with details on password authentication
  • [SPOI-2906] - Write documentation on App Bundle
  • [SPOI-2908] - Prototype kafka-on-yarn
  • [SPOI-2934] - Add application id to the tuple record topic
  • [SPOI-2962] - Create operator to write events in bucket file and write index files for keys
  • [SPOI-2963] - Change query processor operator to fetch events based on indices
  • [SPOI-2996] - Support launch-app-bundle command without the appname argument
  • [SPOI-3006] - Create a LATLON to MGRS converter utility
  • [SPOI-3009] - Create a utility to calculate MD5 hash
  • [SPOI-3010] - Create a utility for Blowfish
  • [SPOI-3014] - Update sandbox release docs
  • [SPOI-3026] - Change the name app bundle to app package and the extension zip to jar
  • [SPOI-3036] - Review issue with ingestion
  • [SPOI-3050] - A first cut at design for ingestion streamlet
  • [SPOI-3051] - License work
  • [SPOI-3108] - Add REST API to get the entire hadoop configuration
  • [SPOI-3121] - HDS - File Format - Performance testing TFile / DTFile
  • [SPOI-2985] - HDS - File Format - Performance testing - HFile
  • [SPOI-2987] - HDS - File Format - Performance testing - MapFile
  • [SPOI-3122] - HDS - File Format - HFile implementation
  • [SPOI-3017] - HDS - File Format - Common Interface
  • [SPOI-3100] - Create the SinglePointCalculator
  • [SPOI-3103] - Create Synchronizer Operator
  • [SPOI-3104] - Create Persister to store the data

Bug

  • [MLHR-1223] - JDBC Store doesn't support connection properties
  • [MLHR-1253] - documenting keyhashvalpair
  • [MLHR-1255] - db api in malhar lib has KeyValueStore interface which is similar to db/cache/Store api
  • [MLHR-1260] - AngularJS Migrations - Metrics Grouping
  • [MLHR-1268] - AbstractBatchTransactionableStoreOutputOperator add to library db is same as public abstract class AbstractAggregateTransactionableStoreOutputOperator
  • [MLHR-1269] - Mobile demo dies after couple of days
  • [MLHR-1319] - AngularJS Migration - WebSocket Service
  • [MLHR-1320] - AngularJS Migration - Page Visibility API
  • [MLHR-1323] - stram events links broken (angular)
  • [MLHR-1325] - Wrong “track by” in physical operators list
  • [MLHR-1328] - “inspect” button in appslist broken
  • [MLHR-1335] - dtPageHref directive does not update href of element on changes
  • [MLHR-1336] - Container shorthand directive not watching value changes
  • [MLHR-1342] - Allow DAG widgets to be vertically resized (using jquery resizable)
  • [MLHR-1344] - AngularJS Migration - Metrics Chart
  • [MLHR-1345] - AngularJS Migration - Physical Operator Page - Chart
  • [MLHR-1346] - Test for AbstractFSDirectoryInputOperator fails sporadically
  • [MLHR-1359] - AngularJS Migration - Dashboard Widgets Vertical Resize
  • [MLHR-1364] - BucketManager while reading values doesn't ensure that latest value for a key is read
  • [MLHR-1368] - AngularJS Migration - App Instance Page - Overview Grouping
  • [MLHR-1374] - Remove all occurrences of dag.setAttribute(DAG.APPLICATION_NAME, “xxx”) in demos code

Improvement

  • [MLHR-1263] - Migrate JDBC non-transaction output operator to new db api and move it to lib
  • [MLHR-1322] - Changing Machine Data Operator and Stream Names
  • [MLHR-1324] - appState directive should be renamed to something more generic
  • [MLHR-1327] - Assume lodash is global throughout console
  • [MLHR-1343] - Use userStorage to store height of stram events
  • [MLHR-1351] - Allow transformResponse to set a fetchError
  • [MLHR-1353] - Centralize location of breadcrumbs
  • [MLHR-1360] - Put dashboards on the left

New Feature

  • [MLHR-419] - DAG Widget with AngularJS
  • [MLHR-1220] - Send the “unsubscribe” webSocket message for topic when no more listeners
  • [MLHR-1239] - AngularJS Migration - Physical DAG View
  • [MLHR-1240] - AngularJS Migration - Logical DAG View
  • [MLHR-1243] - Angular JS Migration - Current/Recover Window Id Metrics
  • [MLHR-1245] - AngularJS Migration - Partitions/Container Metrics
  • [MLHR-1246] - AngularJS Migrations - DAG View Zoom
  • [MLHR-1247] - AngularJS Migration - Physical View
  • [MLHR-1248] - AngularJS Migration - Container View
  • [MLHR-1249] - AngularJS Migration - Landing Page High-Level Metrics
  • [MLHR-1250] - AngularJS Migration - Charts
  • [MLHR-1251] - AngularJS Migration - Gauge Widget
  • [MLHR-1252] - AngularJS Migration - DAG View Base Renderer
  • [MLHR-1278] - AngularJS Migration - App Instance Page - Logical View
  • [MLHR-1279] - AngularJS Migration - App Instance Page - Physical View
  • [MLHR-1280] - AngularJS Migration - App Instance Page - Physical DAG View
  • [MLHR-1283] - AngularJS Migration - App Instance Page - Metric View
  • [MLHR-1298] - AngularJS Migration - Physical Operator Page - Overview
  • [MLHR-1299] - AngularJS Migration - Physical Operator Page - Port List
  • [MLHR-1300] - AngularJS Migration - Physical Operator Page - Properties
  • [MLHR-1303] - AngularJS Migration - Logical Operator Page - Overview
  • [MLHR-1305] - AngularJS Migration - Logical Operator Page - Partitions
  • [MLHR-1306] - AngularJS Migration - Logical Operator Page - Chart
  • [MLHR-1308] - AngularJS Migration - Logical Operator Page - Properties
  • [MLHR-1313] - AngularJS Migration - App Instance Page - Physical View - Container List
  • [MLHR-1314] - AngularJS Migration - App Instance Page - Physical View - Operators
  • [MLHR-1315] - AngularJS Migration - App Instance Page - Logical View - Chart
  • [MLHR-1316] - AngularJS Migration - App Instance Page - Logical View - Stream List
  • [MLHR-1321] - AngularJS Migration - Distribution Build with Gulp
  • [MLHR-1334] - Front-End Build with Gulp - WebSocket Proxy
  • [MLHR-1337] - Front-End Build with Gulp - JS/CSS Revisions
  • [MLHR-1338] - Front-End Build with Gulp - Angular Templates Injection
  • [MLHR-1341] - create userStorage service for user settings
  • [MLHR-1358] - Breadcrumbs to jump between collection elements
  • [MLHR-1361] - Logical/Physical DAG - Stream Locality Legend
  • [MLHR-1365] - AngularJS Migration - Widget Settings Modal - Height Management

Task

  • [MLHR-1208] - Convert contrib apps to app bundles
  • [MLHR-1244] - Update tuple recorder topic to use “applications..tupleRecorder.”
  • [MLHR-1254] - Create a test case for UniqueValueCountAppender
  • [MLHR-1264] - Start Logical Operator Page
  • [MLHR-1265] - Start Physical Operator Page
  • [MLHR-1266] - Start Container Page View
  • [MLHR-1271] - Container Page: overview
  • [MLHR-1273] - Container Page: operator list
  • [MLHR-1275] - Container Page: log viewer
  • [MLHR-1276] - Container Page: page module
  • [MLHR-1284] - Stream View: overview
  • [MLHR-1285] - Stream View: page module
  • [MLHR-1289] - Port View: page module
  • [MLHR-1302] - Operations Landing Page: memory gauge
  • [MLHR-1311] - Vertical Resize (malhar-angular-dashboard)
  • [MLHR-1318] - Move webSocket service to malhar-angular-widgets
  • [MLHR-1326] - Create .jshintrc for IDE/editor, grunt tasks
  • [MLHR-1330] - Document gulp usage in README
  • [MLHR-1333] - add “gulp coverage” task
  • [MLHR-1339] - Need a dynamic partitioner for File processing operator where the count of the operator instances is controlled by the backlog present
  • [MLHR-1348] - Change app bundle to app package on all demos
  • [MLHR-1349] - Link tables to userStorage service
  • [MLHR-1372] - Adding the tool-tip capability to console
  • [MLHR-1376] - Add counters and counters aggregator to mobile demo

Version 1.0.3

Bug

  • [SPOI-2620] - S3 reader error
  • [SPOI-2673] - Move the Kafka Exactly Once Producer to Malhar Library
  • [SPOI-2775] - Log file being processed on seperate port to monitor progress by input operators.
  • [SPOI-2813] - Send top gateways in descending order
  • [SPOI-2814] - Change field name of input event in CDREvent
  • [SPOI-2815] - Show top 3 gateways using a widget
  • [SPOI-2816] - No query poll in alert information view
  • [SPOI-2817] - For non-finalized buckets threshold flags should not be set
  • [SPOI-2819] - Send data continuously to the application
  • [SPOI-2820] - Provide more alerts in the UI
  • [SPOI-2837] - Operators crashing sporadically when alert information requested from frontend
  • [SPOI-2838] - Clear out the cdr event files and aggregates are cleared
  • [SPOI-2844] - Machine data demo waits long before starting
  • [SPOI-2845] - Recovery is failing in cdr events output operator after upstream operator redeploys
  • [SPOI-2854] - Buckets are not being finalized correctly when there is a bucket gap in input data
  • [SPOI-2861] - GATEWAY_CONNECT_ADDRESS not set during install
  • [SPOI-2866] - Change lablel of top gateways to top 3 gateways
  • [SPOI-2867] - Change null to empty value in the UI for alert event information
  • [SPOI-2869] - Need a cleanup script to delete old files from hdfs
  • [SPOI-2882] - Backup the allatory log file for flume integration build as well
  • [SPOI-2883] - NPE in bufferserver
  • [SPOI-2892] - DTGateway WS fails to get YARN logs
  • [SPOI-2894] - Investigate bucket manager for cdr event storage
  • [SPOI-2900] - First few events for an application do not get published
  • [SPOI-2902] - Documentation for sandbox missing images
  • [SPOI-2910] - PartitioningTest.testDynamicDefaultPartitioning sometimes fails and sometimes succeeds
  • [SPOI-2913] - Events API returns no events when limit and offset not supplied
  • [SPOI-2919] - Change cleanup to cleanup folders older than an hour
  • [SPOI-2920] - make stats recorder asynchronous so that it won't block stram
  • [SPOI-2921] - Test Node Locality feature on HDP
  • [SPOI-2932] - Historical containers don't show up on gateway if there is a second attempt on application
  • [SPOI-2936] - Historical data store high level design
  • [SPOI-2937] - Gateway throwing exception while launching app

Improvement

  • [SPOI-2811] - Show input record in alert information dialog
  • [SPOI-2812] - Show top gateways for all defect aggregates and possibly all aggregates
  • [SPOI-2881] - Show actual memory usage in dashboard
  • [SPOI-2891] - DTGateway default log setting too verbose
  • [SPOI-2911] - Create a Bean2String codec

New Feature

  • [SPOI-2228] - Gateway to load and to allow manipulation of app bundles
  • [SPOI-2753] - Front End POC - Aggregates Table
  • [SPOI-2757] - Front End POC - Kafka Operations
  • [SPOI-2758] - Front End POC - Historical Data Navigation
  • [SPOI-2808] - Support YARN log aggregation for dead container log retrieval in the gateway
  • [SPOI-2826] - Maven archetype for assembling app bundle
  • [SPOI-2827] - CLI to load app bundles
  • [SPOI-2828] - Supports launching apps in app bundle in CLI
  • [SPOI-2829] - Gateway to allow uploading and changing configuration of existing App Bundles
  • [SPOI-2849] - Demonstrate Datatorrent RTS as front end ingestion for Spark.
  • [SPOI-2879] - Kafka Request/Response Debug Collapsible Panel
  • [SPOI-2896] - Unique identifier for stram events

Story

  • [SPOI-2835] - Streamlet Design - 1.0.3
  • [SPOI-2836] - Kafka integration for 1.0.3
  • [SPOI-2851] - Platform Excellence - 1.0.3

Task

  • [SPOI-2212] - Determine logistics of application bundles
  • [SPOI-2521] - Add counters for checkpointo operations
  • [SPOI-2546] - Pi demo does not seem to work in 512MB
  • [SPOI-2547] - Twitter demo needs 1024MB
  • [SPOI-2684] - Automated distro certification - collect minimum requirements
  • [SPOI-2786] - Include setters and getters for output fields in the event
  • [SPOI-2795] - Add top 3 gateways to alert aggregate csv in HDFS
  • [SPOI-2802] - Dynamic partitioning for S3 input operator.
  • [SPOI-2822] - Add operator to write cdr events to hdfs
  • [SPOI-2823] - Fetch events from hdfs for alert events query response
  • [SPOI-2846] - Partition Aggregations operator
  • [SPOI-2859] - Separate out query processing from aggregations operator
  • [SPOI-2870] - Application template classpath does not include malhar libraries
  • [SPOI-2874] - Add Port Queue Capacity usage as part of the operator stats
  • [SPOI-2878] - Move the gateway randomization to generator
  • [SPOI-2890] - Update support contacts to malhar-users group
  • [SPOI-2914] - Configure Store operator to join logger and tracker stream.

Sub-task

  • [SPOI-2747] - Total emitted window of an operator keeps on increasing even when it has stopped emitting
  • [SPOI-2850] - Sort out the dependencies for dt-flume so we do not include the jars which are already part of dt dist and hadoop dist
  • [SPOI-2873] - Deprecate ShipContainingJars.
  • [SPOI-2895] - launch -license does not work
  • [SPOI-2907] - Counters appear in physical plan only when a Counters aggregator or a stats listener is set on the operator

Bug

  • [MLHR-1176] - Update demo section of README.md
  • [MLHR-1219] - Non-chrome browsers cannot parse date format from date picker
  • [MLHR-1227] - AbstractKafkaOutputOperator fails with java.lang.NoClassDefFoundError: com/yammer/metrics/Metrics
  • [MLHR-1235] - close button on gateway info modal does not close the modal

Improvement

  • [MLHR-1213] - Use trackBy for malhar-angular-table
  • [MLHR-1216] - DB key value store manager enhancement
  • [MLHR-1218] - Clicking on an open stram log event should close it in UI

New Feature

  • [MLHR-1212] - Make scrollbar draggable on malhar-angular-table
  • [MLHR-1225] - Logical Operator Table
  • [MLHR-1226] - Stram Event widget
  • [MLHR-1229] - Provide better widget options dialog
  • [MLHR-1231] - Custom template/controller for widget options

Task

  • [MLHR-1205] - Restructure directory structure in angular console
  • [MLHR-1206] - Change malhar pom to use provided scope to prevent DT and hadoop jars from being sucked into the runtime classpath
  • [MLHR-1210] - Create script to automatically add scripts to index.html
  • [MLHR-1224] - Kafka enabled AdsDimensions demo

Version 1.0.2

Bug

  • [SPOI-431] - Fix mergeSort operator to actually do mergeSort
  • [SPOI-2499] - Provide license delegation tokens in secure environment
  • [SPOI-2500] - Provide new delegation token before an old one expires in secure enviroment
  • [SPOI-2501] - dt.log not written under CDH5 GA
  • [SPOI-2506] - [MapR]FileSystem.mkdirs() doesn't work for existing folders in MapRFileSystem
  • [SPOI-2605] - DT install as root cannot use /user/chetan/dt0528 as DFS location and not printing out detailed error
  • [SPOI-2608] - LicensingAppMaster's name should not be obfuscated
  • [SPOI-2617] - Memory usage counted by license agent and not released after app is killed
  • [SPOI-2619] - Front End Server - fill missing slots for time series
  • [SPOI-2630] - Not able to launch apps in cloudera cluster
  • [SPOI-2634] - Uninstall is removing entire datatorrent folder as opposed to the release folder only
  • [SPOI-2639] - dtcli allows apps to be launched before configuration is finished
  • [SPOI-2668] - Parent jira for Kafka work for 1.0.2
  • [SPOI-2678] - Put Fraud demo into the demo UI
  • [SPOI-2679] - Front End Server - Query ID as Query JSON
  • [SPOI-2683] - Appmaster Logs are not shown in the Console
  • [SPOI-2689] - set-operator-property produces NPE on stram
  • [SPOI-2699] - Mobile demo app mis-behaving
  • [SPOI-2703] - Gateway fails to start in devel mode with hadoop 2.3.0
  • [SPOI-2705] - Demo UI instructions missing from install README
  • [SPOI-2706] - Logical plan change fails w/ obfuscated build
  • [SPOI-2709] - Partition events should not be registered if same no. partitions result
  • [SPOI-2710] - Containers running the unifier are not released
  • [SPOI-2723] - Design and develop the File Ingestion app
  • [SPOI-2737] - Gateway installation replaces dt-site.xml with invalid version
  • [SPOI-2744] - ZIP of version's docs not being made available for download
  • [SPOI-2748] - dtdemos service fails to stop in sandbox
  • [SPOI-2749] - docs distribution files at root path
  • [SPOI-2759] - Installation instructions for user setup
  • [SPOI-2760] - Get container log content fails with 500
  • [SPOI-2761] - Remove dtadmin reference from install wizzard
  • [SPOI-2762] - EMR configuration issues
  • [SPOI-2779] - dtcli failed to read the license file which was uploaded through UI
  • [SPOI-2798] - Request for log content outside of range results in 500, long request time
  • [SPOI-2839] - set-operator-property broken in master

Improvement

  • [SPOI-2602] - Containers published via websocket should only contain live containers and recently finished containers
  • [SPOI-2610] - Parent: Automatic unobfuscation tool
  • [SPOI-2613] - Create web tool to run unobfuscator
  • [SPOI-2614] - Automate transfer of allatori-log.xml to web server
  • [SPOI-2712] - Add window_width to app info REST call
  • [SPOI-2736] - Merge /systemAlerts/alerts and /systemAlerts/inAlerts API calls
  • [SPOI-2751] - Include sandbox README.html with docs.zip and on website

New Feature

  • [SPOI-950] - Specify memory requirements on per operator basis (duplicate)
  • [SPOI-2220] - Gateway App Bundle API spec
  • [SPOI-2339] - Enable container size for each operator(s)
  • [SPOI-2515] - Ability to dynamically change the logger level for any instantiated loggers within the application
  • [SPOI-2523] - System alerts for application state and metrics
  • [SPOI-2525] - Add operator serialization check in local mode
  • [SPOI-2528] - Expose “counters” in REST API
  • [SPOI-2575] - Kafka Pub/Sub Protocol
  • [SPOI-2588] - Record physical counters per window
  • [SPOI-2589] - Expose counters logical and physical through web services/ web socket
  • [SPOI-2606] - Initializing loggers with levels specified in the configuration
  • [SPOI-2627] - Front End Server - “Countdown” usage
  • [SPOI-2629] - Front End Server - Support of Multiple Partitions
  • [SPOI-2631] - Front End Server - Cache Expiration Strategy
  • [SPOI-2632] - Front End Server - Kafka Reconnect
  • [SPOI-2633] - Front End Server - Page Performance (Visibility API)
  • [SPOI-2638] - Front End Server - Kafka Errors Handling
  • [SPOI-2645] - Front End Server - Kafka SimpleConsumer
  • [SPOI-2652] - Develop and Stage Hadoop Summit Landing Page
  • [SPOI-2696] - Front End Server - Top N Metrics
  • [SPOI-2697] - Front End Server - Dynamic Publisher and Site
  • [SPOI-2698] - Front End Server - Kafka Troubleshooting
  • [SPOI-2752] - Front End Server - LRU Cache
  • [SPOI-2756] - Front End POC - Alert Modal
  • [SPOI-2804] - Create a hbase operator that uses a config to map incoming csv tuples to hbase table data and saves them in hbase

Story

  • [SPOI-2455] - HDFS storage layer for 1.0.2
  • [SPOI-2565] - Front End Server - Define cache policy for push data in Node.js
  • [SPOI-2571] - Parent jira for Hadoop Summit work
  • [SPOI-2590] - Parent: GA Testing
  • [SPOI-2611] - Parent jira for facilitating trouble shooting and debugging in 1.0.2
  • [SPOI-2615] - Parent jira for security work in 1.0.2
  • [SPOI-2662] - Parent jira for licensing security for 1.0.2
  • [SPOI-2664] - HDFS Storage, Ingestion, Access application data and file storage for 1.0.2
  • [SPOI-2667] - Automate distro certification and make it part of CI for 1.0.2
  • [SPOI-2781] - Determine which fields are present in output that are not present in input

Task

  • [SPOI-2341] - Update on authentication and authorization in Yarn-open source
  • [SPOI-2434] - Short document on deduper checkpointing scheme
  • [SPOI-2440] - Kafka support in POC for 1.0.2
  • [SPOI-2509] - Investigate how to use Kafka to replace the pub/sub mechanism in Gateway
  • [SPOI-2532] - Join Operator
  • [SPOI-2534] - Add support of getting container info of dead applications and dead containers
  • [SPOI-2535] - Add support for retrieving Aggregated Counters from Response processStats of StatsListener
  • [SPOI-2542] - Get user to look at instructions before .bin file download
  • [SPOI-2544] - Twitter demo “feedMultiplier” should be RW and test to ensure that it can be changed in runtime
  • [SPOI-2548] - Memory gauge always shows 100%
  • [SPOI-2549] - Total memory on right corner needs to be discussed
  • [SPOI-2552] - Need to list certified Distros on site
  • [SPOI-2558] - Help set up Kafka operations
  • [SPOI-2568] - Making Kafka Producer Exactly Once
  • [SPOI-2569] - Add the Rewind Feature
  • [SPOI-2573] - Integrate the Scoring of the quality logs into the DAG
  • [SPOI-2574] - Add counter calculation in hdfs operator
  • [SPOI-2584] - DT counters as Key,Val (String,Number)
  • [SPOI-2585] - Troubleshooting work for reporting min, max, ave, ... on resources
  • [SPOI-2591] - GA Testing: CDH5 end-to-end
  • [SPOI-2592] - GA Testing: HDP2 end-to-end
  • [SPOI-2593] - GA Testing: MapR end-to-end
  • [SPOI-2594] - GA Testing: UI end-to-end
  • [SPOI-2595] - GA Testing: dtcli / gateway end-to-end
  • [SPOI-2596] - GA Testing: Apache Hadoop end-to-end
  • [SPOI-2597] - GA Testing: High Availability / Recovery
  • [SPOI-2598] - GA Testing: Application configuration and launch
  • [SPOI-2599] - GA Testing: Sandbox functionality
  • [SPOI-2600] - GA Testing: Sandbox UX
  • [SPOI-2621] - Top 10 support
  • [SPOI-2622] - Partitioning of DimensionStore operator
  • [SPOI-2624] - Front End Server - Architecture
  • [SPOI-2626] - Setting up of number of partitions for Kafka Producer
  • [SPOI-2636] - Twitter demo app with hashtag top 10
  • [SPOI-2640] - Setup UI server for HDP grid for Hadoop Summit
  • [SPOI-2641] - Setup Ambari for Hadoop Summit Demo
  • [SPOI-2642] - Run Twitter demo on HDP cluster for Hadoop Summit
  • [SPOI-2643] - Setup Mobile demo for HDP demo at Hadoop Summit
  • [SPOI-2644] - Set up machine data demo on HDP cluster for Hadoop Summit
  • [SPOI-2646] - Setup Ads demo on HDP cluster for Hadoop Summit
  • [SPOI-2647] - Add the new Twitter HashTag top 10 demo to Frontend server for Summit
  • [SPOI-2650] - Change default license memory settings
  • [SPOI-2651] - Milestone 1 update
  • [SPOI-2670] - Launch twitter hashtags on CDH and HDP clusters
  • [SPOI-2671] - Update CDH cluster DT UI
  • [SPOI-2672] - Move the HDFSOutputOperator to Malhar Library
  • [SPOI-2676] - Get events, tuple records from kafka
  • [SPOI-2680] - Make more managable kafka cluster
  • [SPOI-2682] - Customer Demos Support
  • [SPOI-2685] - Automated distro certification - virtualization review
  • [SPOI-2686] - Automated distro certification - provisioning review
  • [SPOI-2687] - Certify DT on Pivotal HD
  • [SPOI-2688] - Make twitter HashTags links in demo
  • [SPOI-2690] - Upgrade security sections in guides
  • [SPOI-2700] - Test DimensionStore operator checkpointing and recovery
  • [SPOI-2714] - Document stram event types
  • [SPOI-2732] - Create a multi level key map with multiple keys
  • [SPOI-2733] - Create a CSV lookup class to load CSV and lookup values based on row keys
  • [SPOI-2735] - Ensure all dimemsions are implemented and covered by test
  • [SPOI-2763] - Create a csv parser framework that can handle the different csv file formats
  • [SPOI-2764] - Create a template application
  • [SPOI-2765] - Create the template configuration file
  • [SPOI-2766] - Create kafka input for the application
  • [SPOI-2767] - Add a directory scan operator that progressively scans for new folders and files for input data to the application
  • [SPOI-2768] - Add a kafka upload script to upload data to the application
  • [SPOI-2769] - Create an upload script that uploads application files to hdfs
  • [SPOI-2773] - Add a property to clear the aggregates in the AggregationsOperator
  • [SPOI-2785] - Include the input event string in the CDR event
  • [SPOI-2789] - Add query support for getting events for alerts
  • [SPOI-2790] - Scale the input events into millions
  • [SPOI-2792] - Send top 3 gateway list in alert event details
  • [SPOI-2793] - Add the capability to finalize a bucket
  • [SPOI-2794] - Add ability to clear out old results when replaying data
  • [SPOI-2797] - Add query support to send the latest five minute buckets and aggregates
  • [SPOI-2800] - Install a kafka web-console in our cluster
  • [SPOI-2805] - Investigate yarn log aggregation to see how this fits into our container logs API

Sub-task

  • [SPOI-2725] - Create skeleton App
  • [SPOI-2728] - Convert the Skeleton Operators into Real Operators - Part I
  • [SPOI-2739] - File input operator
  • [SPOI-2740] - DAG level unit test
  • [SPOI-2776] - Test and Certify RPM packaging for Cloudera
  • [SPOI-2777] - Gateway throws a bunch of exceptions when trying to install using rpm

Bug

  • [MLHR-724] - RabbitMQ test timing issue
  • [MLHR-1128] - Breadcrumbs should say physical operator, not operator on port page
  • [MLHR-1129] - DAG should not display underneath controls
  • [MLHR-1137] - Memory gauge always shows 100%
  • [MLHR-1142] - Front End Server - Kafka Reconnect
  • [MLHR-1143] - KafkaOutputOperator is not configurable for Kafka Producer properties
  • [MLHR-1144] - Container Actions - use wording that's not confusing
  • [MLHR-1157] - Parent JIRA for Front End Server
  • [MLHR-1159] - Stop Recording function does not give visual indication of stopped recording unless refreshed
  • [MLHR-1169] - Update readme for console repo
  • [MLHR-1170] - Appmaster Logs are not shown in the Console
  • [MLHR-1185] - Stram Event widget should check physical plan for existence of operator
  • [MLHR-1186] - some numbers in logical dag hidden behind control
  • [MLHR-1189] - AbstractHdfsTupleFileOutputOperator output port is not optional
  • [MLHR-1192] - Parametrize the demos
  • [MLHR-1195] - App master container is not showing proper metrics
  • [MLHR-1202] - failureCount missing for physical operators

Improvement

  • [MLHR-1132] - Improvements to errors in the stram events widget
  • [MLHR-1146] - Only get active containers for initial container list
  • [MLHR-1167] - UI support for searching log levels of classes
  • [MLHR-1168] - Validate input for log level widget
  • [MLHR-1194] - Align app counts with labels in cluster overview widget
  • [MLHR-1209] - Support JMS providers other than ActiveMQ

New Feature

  • [MLHR-921] - Installer - List of Errors/Codes
  • [MLHR-1111] - Container Log Widget, using new REST call
  • [MLHR-1121] - Front End Server - Kafka Query JSON
  • [MLHR-1122] - Front End Server - Kafka Keep Alive
  • [MLHR-1123] - Front End Server - Kafka Parameterized Queries
  • [MLHR-1124] - Front End Server - Kafka Response Debug/Format
  • [MLHR-1125] - Front End Server - Kafka Pub/Sub Protocol for Widgets
  • [MLHR-1130] - Widget to change the log levels of classes dynamically
  • [MLHR-1133] - Put “float” control in widget config popup
  • [MLHR-1134] - Front End Server - Editable JSON request
  • [MLHR-1135] - Front End Server - Setup Instructions
  • [MLHR-1136] - Front End Server - Packaging/Distribution
  • [MLHR-1147] - Front End Server - Selecting Metrics
  • [MLHR-1149] - Front End Server - Use Latest Kafka Offset
  • [MLHR-1151] - Front End Server - Data Validation
  • [MLHR-1152] - Front End Server - Page Performance (Visibility API)
  • [MLHR-1175] - Widget Library - Select Widget
  • [MLHR-1179] - AngularJS Migration - Application Instance Page
  • [MLHR-1181] - AngularJS Migration - Application Overview Widget
  • [MLHR-1182] - Angular JS Migration - App Structure
  • [MLHR-1187] - UI for adding system alerts
  • [MLHR-1193] - Create Basic Counters and its aggregator in library

Story

  • [MLHR-1178] - AngularJS Migration for 1.0.2

Task

  • [MLHR-960] - Create a Tableau output adapter
  • [MLHR-1090] - Create first resource objects (models) for angular console
  • [MLHR-1106] - Improve ended app page
  • [MLHR-1139] - Add HTTP Get Operator
  • [MLHR-1140] - Add HDFS output operator that writes to file names specified in tuple
  • [MLHR-1160] - Create HDFS Output Operator to write with exactly once semantics
  • [MLHR-1162] - Implement app list with angular
  • [MLHR-1164] - Wireframe for UI DAG builder
  • [MLHR-1171] - Make twitter hashTab a link in twitter demo
  • [MLHR-1172] - Create a Cassandra read operator
  • [MLHR-1173] - Create a Cassandra write operator

Sub-task

  • [MLHR-1163] - Wireframe for DAG builder

Version 1.0.1

Bug

  • [SPOI-1404] - Create seperate environment for node0 for demos
  • [SPOI-1952] - Operation and Install guide review
  • [SPOI-2203] - Configure adsDimesions and machine data to use minimum memory
  • [SPOI-2340] - License Agent is persisting state in a wrong location
  • [SPOI-2397] - Monitoring and fixing the bugs in Customer demos
  • [SPOI-2408] - Verify DT Platform on HDP 2.1
  • [SPOI-2410] - Master script to restart hadoop on the cluster
  • [SPOI-2411] - Not able to starte license with 0.9.5 release
  • [SPOI-2420] - Core doesn't compile with hadoop 2.4
  • [SPOI-2448] - Test xml ingestion demo on a cluster
  • [SPOI-2449] - Test golden gate demo on a cluster
  • [SPOI-2450] - Test twitter sentiment demo on a cluster
  • [SPOI-2474] - Containers not provisioned diagnostics error
  • [SPOI-2477] - LicenceAgent client fails with NPE
  • [SPOI-2489] - Operator downstream of EXACTLY_ONCE operator not checkpointing
  • [SPOI-2490] - HDFS
  • [SPOI-2491] - Testing the HDFS drain for Flume Sink
  • [SPOI-2493] - Add HDFS operator updates into Malhar
  • [SPOI-2496] - viewdag does not work any more
  • [SPOI-2503] - Provide a rewind feature to replay tuples from a previous time
  • [SPOI-2505] - Sandbox demos need customization
  • [SPOI-2512] - Apps fail to start due to serialization errors on CDH5
  • [SPOI-2533] - Fraud app needs debugging
  • [SPOI-2543] - Applications not reporting the license information to License Agent if LA is restarted
  • [SPOI-2550] - License Agent Dying
  • [SPOI-2554] - Third party libraries which are required for demos are missing from /opt/datatorrent/releases/version/lib/
  • [SPOI-2556] - Add demo guides to documentation index

Improvement

  • [SPOI-2210] - Default license generation should be part of build process
  • [SPOI-2468] - README for certifications and benchmarks
  • [SPOI-2482] - DTCli should handle ^C more gracefully
  • [SPOI-2494] - Add HTTP Get operator to Malhar
  • [SPOI-2555] - Include README, LICENSE, and CHANGELOG in docs

New Feature

  • [SPOI-406] - Document and test download to work with Hortonworks HDP 2.0
  • [SPOI-408] - Certify with MapR (Hadoop 2.3) release in 1.0.1
  • [SPOI-1146] - API for retrieving raw Container Logs
  • [SPOI-1834] - Pub-Sub/REST mechanisms for exceptions thrown by an application
  • [SPOI-2080] - Application to certify installation
  • [SPOI-2253] - Username/password feature in the installer
  • [SPOI-2381] - dt-site.xml logistics
  • [SPOI-2387] - Aggregate Logical Operators on Gateway
  • [SPOI-2402] - Update dt-site.xml with container memory less than 512MB after getting the app-memory-automation results for all the apps.
  • [SPOI-2460] - dt-site.xml new logistics implementation
  • [SPOI-2470] - Create a certification property file for Distro/Install Certification
  • [SPOI-2486] - Workaround for “FileSyste.getScheme is not overrided by MapRFileSystem”
  • [SPOI-2519] - add grep feature when getting container log
  • [SPOI-2526] - Add REST call to return info of historical containers

Task

  • [SPOI-62] - Buffer Server logging
  • [SPOI-1206] - Review Licensing of Third Party Libraries
  • [SPOI-1745] - Document Demo Application Setup
  • [SPOI-1784] - License agent app to have webservice
  • [SPOI-2199] - Readme file for AWS installation
  • [SPOI-2223] - Test if all the applications with reduced container memory sizes run in sandbox 0.9.4.
  • [SPOI-2348] - Enable checkpoint for aggregator operator
  • [SPOI-2354] - Certify demos on 0.9.4 sandbox
  • [SPOI-2370] - Fully automate Hadoop cluster restart
  • [SPOI-2379] - Certify DT with Hadoop 2.3/CDH
  • [SPOI-2392] - Add more basic demos to sandbox launcher
  • [SPOI-2393] - Certify Amazon EMR Hadoop 2 version
  • [SPOI-2414] - Stop the nightly benchmarking jenkins jobs
  • [SPOI-2415] - remove the “page has not loaded since 60 second” message at top
  • [SPOI-2422] - Add HDFS cache limit on FlumeAgent operator
  • [SPOI-2424] - Log metrics across all poc operators to a file
  • [SPOI-2425] - Compare HDFS with customer site
  • [SPOI-2426] - Benchmark HDFS drain rate by agent operator
  • [SPOI-2427] - Enable dedupper to be run with checkpointing turned OFF
  • [SPOI-2428] - Evaluate bigger bucket size for checkpointing for deduper
  • [SPOI-2430] - Number validation for poc
  • [SPOI-2432] - Compute the checkpoint I/O for dedupper
  • [SPOI-2439] - Compute deduper HDFS I/O for current checkpointing scheme
  • [SPOI-2441] - Estimate the feed time for HDFS agent replay
  • [SPOI-2445] - Certify Cloudera CDH 5.0 GA
  • [SPOI-2451] - Gateway Load Test
  • [SPOI-2457] - Widget Library
  • [SPOI-2458] - Generate certification report for HDP 2.1
  • [SPOI-2466] - Monitoring and notifications for cluster
  • [SPOI-2475] - Document process to certify demo re-start
  • [SPOI-2476] - Document process for grid restart
  • [SPOI-2483] - Node.js Kafka Client POC
  • [SPOI-2513] - Create demos ui service wrapper compatible with CentOS
  • [SPOI-2517] - Configure customer demos ui for as a service
  • [SPOI-2531] - HDFSOutputOperator write with exactly once semantics
  • [SPOI-2535] - Add support for retrieving Aggregated Counters from Response processStats of StatsListener
  • [SPOI-2537] - Need to list previous versions on datatorrent site
  • [SPOI-2540] - Fix URL for AWS EMR document
  • [SPOI-2551] - Define a process on backend to ensure archive of previous downloads works for customers
  • [SPOI-2553] - Holding ticket for Malhar maintenance work

Sub-task

  • [MLHR-774] - Terms of service license
  • [MLHR-976] - Implement table module for angular

Bug

  • [MLHR-998] - Compilation error while using UniqueValueCount operator.
  • [MLHR-1054] - Update topic and rest calls for Logical/Physical Split
  • [MLHR-1056] - Rename demo apps to include Demo suffix
  • [MLHR-1089] - Error handling for license page
  • [MLHR-1093] - AbstractKafkaInputOperator is not committing the txn to Kafka in commit call back
  • [MLHR-1096] - Get last N events when tailing stram events
  • [MLHR-1108] - Logical DAG resizing
  • [MLHR-1113] - AdsDimension demo fails to launch
  • [MLHR-1114] - FraudDetect demo fails to launch
  • [MLHR-1115] - MachineData demo fails to launch
  • [MLHR-1127] - HdfsTextFileInputOperator Fails during checkpointing
  • [MLHR-1138] - Remove unused CPU columns in logical operator table

Improvement

  • [MLHR-1092] - Add environment variable override to rename demo apps
  • [MLHR-1098] - Improved code coverage for malhar-angular-dashboard
  • [MLHR-1099] - Extend dataModelOptions with non-serializable defaults
  • [MLHR-1100] - Improved widget directive in malhar-angular-dashboard
  • [MLHR-1116] - Use negative offset to get last N events for stram events widget
  • [MLHR-1119] - Add a confirm box when killing app master container
  • [MLHR-1120] - Save range selection between page loads for stram events

New Feature

  • [MLHR-783] - DAG should adjust in height as the widget height changes
  • [MLHR-784] - WidgetOutputOperator - Schema Update
  • [MLHR-984] - STRAM decisions widget
  • [MLHR-1070] - Web Demos - Upgrade to AngularJS 1.2.16
  • [MLHR-1076] - Widget Library - WebSocket
  • [MLHR-1077] - Widget Library - Data Models
  • [MLHR-1087] - Dashboard Component - AngularJS Scope Documentation
  • [MLHR-1088] - Dashboard Web App - Dependencies Documentation
  • [MLHR-1095] - Node.js Kafka Integration
  • [MLHR-1097] - Implement storage on malhar-angular-dashboard
  • [MLHR-1118] - Add storage hash for clearing out invalid dashboards

Story

  • [MLHR-719] - Time Series Forecasting
  • [MLHR-1060] - Widget Library
  • [MLHR-1061] - Dashboard Component
  • [MLHR-1082] - Parent jira for Real Time ETL Application

Task

  • [MLHR-700] - Develop operator for calculating Coefficient of Determination (RSquare)
  • [MLHR-841] - Demo guide for machine data application
  • [MLHR-850] - Demo guide for Ads Dimension demo
  • [MLHR-852] - Demo guide for Twitter (rolling topwards) demo application
  • [MLHR-874] - Demo guide for charting demo - Yahoo finance
  • [MLHR-896] - widget output operator - widget type should not be decided in backend
  • [MLHR-950] - Create a demo for distributed distinct
  • [MLHR-953] - Set up “upgrade path” for CustomerApplications
  • [MLHR-983] - UI to expose the “events” API of the gateway
  • [MLHR-1042] - ETL: Consolidate output operators properties
  • [MLHR-1063] - Create contributing section on the main page of malhar repo
  • [MLHR-1064] - Create contributing secion on main page of UI repos
  • [MLHR-1065] - create line chart operator
  • [MLHR-1075] - add .travis.yml file to ui-console
  • [MLHR-1078] - Update license header script, keep year updated
  • [MLHR-1080] - create real time chart output operator
  • [MLHR-1083] - Consolidate realtime output operator properties
  • [MLHR-1084] - Consolidate aggregate operator properties
  • [MLHR-1085] - Consolidate linechart operator properties
  • [MLHR-1104] - Design App Bundle and Upload/Launch Feature

Version 1.0.0

Bug

  • [SPOI-2086] - creation of logs directory under .dt folder of user
  • [SPOI-2497] - Remove hard-coded “hdfs” scheme
  • [SPOI-2501] - dt.log not written under CDH5 GA
  • [SPOI-2577] - After installation, I cannot launch license agent
  • [SPOI-2604] - The example DFS directory should be changed from /home/... to /user/...
  • [SPOI-2607] - Install page should only list error (not warnings) issues
  • [SPOI-2609] - When yarn.scheduler.minimum-allocation-mb is set to 256, PiDemo fails with NPE

Improvement

  • [SPOI-2522] - Replace InputPort.getStreamCodec method with an attribute on InputPort called StreamCodec
  • [SPOI-2576] - Change local install from .datatorrent to datatorrent directory
  • [SPOI-2603] - installer complains about gateway has trouble starting but it actually runs OK
  • [SPOI-2637] - Improve the Getting Started Guide

Task

  • [SPOI-2520] - Change the name of the CustomStats to Counters
  • [SPOI-2578] - Check dir before mkdir call (MapR requirement)
  • [SPOI-2579] - Take out launch from gateway webservice spec document
  • [SPOI-2649] - Update software license

Bug

  • [MLHR-1101] - Install errors formatting
  • [MLHR-1103] - Change installer instructions for DFS directory path

Version 0.9.5.1

Bug

  • [SPOI-2554] - Third party libraries which are required for demos are missing from /opt/datatorrent/releases/version/lib/
  • [MLHR-1113] - AdsDimension demo fails to launch
  • [MLHR-1114] - FraudDetect demo fails to launch
  • [MLHR-1115] - MachineData demo fails to launch

Version 0.9.5

Sub-task

  • [SPOI-1692] - Certify fault tolerance for multiple containers failure
  • [SPOI-1693] - Develop automated test scenario for StrAM failure to certify one of high availability aspects.
  • [SPOI-2008] - Setup virtual cluster for high availability certification
  • [SPOI-2009] - WordCount app to be used in certification of high availability
  • [SPOI-2127] - Error/Warning message not clear - nightly build installer (datatorrent-dist-0.9.4-SNAPSHOT.bin) if executed as non-root user.
  • [SPOI-2128] - Nightly build installer needs to check if port 9090 is available before attempting to run dtgateway service.
  • [SPOI-2129] - Error/Warning message not clear while updating the Hadoop configuration through Installer-Web-based UI
  • [SPOI-2247] - Gateway user authentication REST API spec
  • [SPOI-2248] - Implement user authentication and basic authorization in gateway
  • [SPOI-2255] - Password protect the web socket
  • [SPOI-2257] - Support HTTPS in gateway
  • [SPOI-2277] - Support HTTPS and auth in pubsub connection from stram to Gateway
  • [SPOI-2291] - Copy GW configurations to HDFS on every write

Technical task

  • [SPOI-2005] - Verify and add Machine demo to app memory usage automation

Bug

  • [SPOI-1801] - Mobile app has two logical input gen - Need to handle containers on servers with two diff clocks
  • [SPOI-1890] - Application behavior when resources are not available
  • [SPOI-1954] - Incorrect Processed Stats
  • [SPOI-1995] - Sandbox instructions are outdated
  • [SPOI-1998] - Killing of node manager on node running License AM
  • [SPOI-2067] - Move apps from contrib to demos
  • [SPOI-2152] - Stabillity of mobile demo
  • [SPOI-2194] - Gateway start script prints out repetitive message
  • [SPOI-2205] - Make generate license tool work with web tool to generate 1TB license in the new flow
  • [SPOI-2207] - Deduper showed high latency and crashed in staging environment.
  • [SPOI-2221] - Analyze FileSystem.get calls and if necessary replace them with FileSystem.newInstance
  • [SPOI-2278] - Relative hdfs path should be created under application directory
  • [SPOI-2283] - Do not make assumption about FSStorageAgent being the StorageAgent or the default configuration for it being the configuration the user set
  • [SPOI-2284] - User ‘dtadmin’ should be added to the group having access to hadoop services and hdfs
  • [SPOI-2285] - uninstall.sh does not uninstall the platform
  • [SPOI-2288] - NullPointerException in buffer server
  • [SPOI-2289] - FSStorageAgent store throws java.nio.channels.ClosedChannelException
  • [SPOI-2295] - Monitoring and fixing the bugs in Customer demos
  • [SPOI-2297] - Mobile Demo going to unstable state due to out of order tuple sequence error
  • [SPOI-2298] - Events returned from REST API has funky keys
  • [SPOI-2302] - Normalize WebSocket & REST format for Stram Events
  • [SPOI-2314] - Fix application names
  • [SPOI-2315] - node1 application launch-xxx macros not working after upgrade
  • [SPOI-2342] - DTCLI ignoring annotated Application Names
  • [SPOI-2344] - dtcli fails to load dt-env.sh
  • [SPOI-2349] - Update default license to 4 months
  • [SPOI-2357] - Unable to launch application using launch alias, if the application names shown in the dtcli list are picked from dt-site.xml.
  • [SPOI-2358] - Installer bin extrac does not preserve permissions
  • [SPOI-2359] - Gateway fails to restart due to missing JAVA_HOME
  • [SPOI-2361] - After global install local .dt directory may be missing when executing dtcli
  • [SPOI-2364] - Catastrophic Error- tuples out of sequence in Generic Node
  • [SPOI-2365] - Hadoop Shell throws exception for dt.attr.APPLICATION_PATH
  • [SPOI-2371] - DTCLI updating the dt-site.xml
  • [SPOI-2373] - Sandbox dt-site.xml is missing configurations related to Map reduce applications.
  • [SPOI-2374] - The apps in sandbox use 1GB as the container size and not 512MB as specified under misc/sandbox/conf/dt/dt-site.xml
  • [SPOI-2375] - configuration from dt-site.xml in user's directory is not picked by dtcli
  • [SPOI-2376] - Gateway classpath references invalid directory
  • [SPOI-2380] - get-physical-operator-properties wrong help text
  • [SPOI-2389] - DataTorrent discovers override dt-site.xml in CLASSPATH
  • [SPOI-2390] - DTGateway is killed after terminal is closed
  • [SPOI-2391] - dtdemos service does not stop on sandbox
  • [SPOI-2395] - Missing FraudDetection/AdsDeminsions/MachineData demos from sandbox.
  • [SPOI-2396] - gateway not started after stopping and starting hadoop/datatorrent services in sandbox.
  • [SPOI-2398] - Extra directory created by gateway in user home
  • [SPOI-2400] - An operator exits normally when there is OutOfMemory error
  • [SPOI-2401] - The AppMaster finishes after retrying 6 times to deploy input operator
  • [SPOI-2412] - App restart HDFS file permission issue on CDH cluster
  • [SPOI-2417] - DTGateway dt-site.xml backup location
  • [SPOI-2419] - Ensure principal name propagates to containers in non-secure mode
  • [SPOI-2435] - Latest changes to latency calculation break tests.

Improvement

  • [SPOI-1376] - Have app specific configuration to stram-site.xml in sandbox
  • [SPOI-1855] - Ability to continue/shutdown application
  • [SPOI-1986] - In Logical View do not include the unifier stats
  • [SPOI-2001] - Update Release Process document (GDrive)
  • [SPOI-2222] - Constraint format in license file to be in JSON
  • [SPOI-2287] - Allow specifying a complete list of dependencies while deploying the application
  • [SPOI-2335] - Make BaseOperator Java Serializable
  • [SPOI-2367] - Create service wrapper for demos server
  • [SPOI-2385] - Improve the performance of HDFSStorage for POC
  • [SPOI-2404] - Reset the Application Failure Count upon successful recovery
  • [SPOI-2405] - Add more params to /ws/v1/applications/{appid}/events call
  • [SPOI-2407] - Need ability to supply the class name representing the property

New Feature

  • [SPOI-91] - Design and implement a socket adapter operator
  • [SPOI-2007] - Automation to certify High Availability
  • [SPOI-2012] - Run the app-memory-usage with lower container sizes starting from 128M to 512M when the grid is restarted.
  • [SPOI-2249] - Simple user authentication / authorization support in the UI
  • [SPOI-2355] - Add REST call to make existing license file the current license
  • [SPOI-2386] - REST call to retrieve the datatorrent conf directory
  • [SPOI-2399] - Starter Web Application

Story

  • [SPOI-2215] - Password protect the dashboard

Task

  • [SPOI-746] - DT Phome home customer side work
  • [SPOI-1642] - Certify support for node failure
  • [SPOI-2045] - Discuss license cleanup/improvements
  • [SPOI-2099] - Show mandatory fields with ‘*’ in the installer
  • [SPOI-2101] - Say “Terms of Service” on the text
  • [SPOI-2103] - Show error message on the side
  • [SPOI-2117] - An error during uninstall
  • [SPOI-2118] - 1 yr license mail not getting mailed
  • [SPOI-2145] - Set up separate access to jars for big customers
  • [SPOI-2178] - Filters HashMap from HDFSLoader
  • [SPOI-2190] - Create web page (navigator and table view) for 5 dimensions and 2 computations
  • [SPOI-2201] - Create a upgrade license page
  • [SPOI-2202] - Upgrade Machine Data to latest code
  • [SPOI-2211] - Getting too many deprecated warnings in demos
  • [SPOI-2213] - Be able to diagnose Stram decisions
  • [SPOI-2214] - Ability to access Stram decisions for failed/finished/killed app
  • [SPOI-2216] - Upgrade Twitter Demo to latest Code
  • [SPOI-2217] - Upgrade Fraud Detection Demo to latest Code
  • [SPOI-2218] - Upgrade Ads Dimension Demo to latest Code
  • [SPOI-2219] - Upgrade Mobile Locator Demo to latest Code
  • [SPOI-2224] - Fraud demo issues
  • [SPOI-2225] - Issues with Machine Data Demo
  • [SPOI-2226] - Issues with Ads demo
  • [SPOI-2233] - Remove max num container parameter that is included by default
  • [SPOI-2234] - Give warning when operators are automatically inlined
  • [SPOI-2237] - CPU column on logical view
  • [SPOI-2243] - Use hammerdb to generate load against Oracle
  • [SPOI-2256] - Document certification suite used to certify datatorrent platform
  • [SPOI-2286] - Implement Goldengate java handler to capture db change and send to kafka
  • [SPOI-2290] - Hide configuration and launch app for GA
  • [SPOI-2292] - Add to README: not recommended to launch more than one gateway
  • [SPOI-2293] - Manage license file on HDFS
  • [SPOI-2303] - Make DFS error keys more fine grained
  • [SPOI-2309] - Use connection to RM to guess the cluster accessible IP address for GATEWAY_CONNECT_ADDRESS
  • [SPOI-2310] - Allow different filesystem from fs.defaultFS for dt.dfsRootDirectory
  • [SPOI-2338] - The “#” in the boxes should not change
  • [SPOI-2351] - Bundle the default passwd file (with admin/admin) in the installer
  • [SPOI-2362] - Create service wrappers for Apache Hadoop on Sandbox
  • [SPOI-2363] - Update dtdemo sandbox script to work with Hadoop/DTGateway service wrappers
  • [SPOI-2394] - Install wizard shows up again after gateway restart
  • [SPOI-2416] - Separate malhar and malhar-ui-console
  • [SPOI-2418] - Create an ability for the DTFlumeSink to backoff if it suspects that the DAG is not processing the data in healthy fashion.
  • [SPOI-2421] - Update end-user documentation

Sub-task

  • [MLHR-726] - Time Series Forecasting Operator using Holt's Linear Trend Model
  • [MLHR-933] - Time Series Forecasting using Holt-Winters' Seasonal Method
  • [MLHR-965] - create non transactional output operator
  • [MLHR-966] - create transactional output operator
  • [MLHR-967] - create a data store writer implementation to use in real time etl app
  • [MLHR-1036] - Create incremental model for Holt's Linear Trend Forecasting Algorithm
  • [MLHR-1040] - Develop incremental model creation for Holt-Winters' Multiplicative Method Time-Series Forecasting

Bug

  • [MLHR-778] - Logical Operators list has no palette
  • [MLHR-857] - Machinedata demo stop working with new Redis operator
  • [MLHR-963] - create output operator to add historical data to supplied datastore
  • [MLHR-964] - create interface for data stores that can be used by output operator to persist historical data
  • [MLHR-968] - Remove test related code from AbstractTransactionableStoreOutputOperator
  • [MLHR-969] - Create a manageable version of our demo docs to be included with our installation
  • [MLHR-978] - create resource leak for the hdfs input/output operators
  • [MLHR-979] - AdsDimension: Redis operator shows high latency
  • [MLHR-986] - Page Not Found errors from the web-site
  • [MLHR-1000] - Invalid IP address selected by default during upgrade
  • [MLHR-1005] - CPU column on logical view
  • [MLHR-1008] - twitter application failed with exception
  • [MLHR-1015] - State is missing from physical operator overview widget
  • [MLHR-1019] - Make gateway ip and port inputs on same line in installer
  • [MLHR-1020] - License screen does not reflect new license upload
  • [MLHR-1021] - Unifiers should not have link to nonexistent logical operator
  • [MLHR-1022] - Update installer license text to reflect correct trial period
  • [MLHR-1028] - Cannot kill application in state ACCEPTED
  • [MLHR-1030] - Sorting on logical operators list fails until you sort on name
  • [MLHR-1031] - UI Showing wrong total Memory usage
  • [MLHR-1049] - Streams not showing up in dag overview
  • [MLHR-1050] - Remove the “Development” mode
  • [MLHR-1055] - Unnecessary gateway restart requested during installation navigation
  • [MLHR-1057] - Overview in dag widget not accurate to visible area
  • [MLHR-1058] - Change install wizard from widget in a page to just a page
  • [MLHR-1066] - Install Wizard tab order and auto-focus
  • [MLHR-1067] - Install Wizard invalid DFS directory check
  • [MLHR-1072] - Socket input operator test fail
  • [MLHR-1074] - Remove stram events page/widget from 0.9.5

Improvement

  • [MLHR-721] - Migrate database cache lookup to the new db framework
  • [MLHR-985] - AdsDimension: Plug-in Dimension Unifier
  • [MLHR-1010] - Remove info icon from installer text
  • [MLHR-1011] - Change “property” to “item” in install step tables
  • [MLHR-1012] - Review and edit installer text
  • [MLHR-1013] - Installer: Rework Hadoop and System screens
  • [MLHR-1035] - Remove darker grey background from console
  • [MLHR-1046] - Operators/Containers not being subscribed to on App Page
  • [MLHR-1062] - Install Wizard examples of Hadoop and DFS path
  • [MLHR-1069] - License summary section in Install Wizard

New Feature

  • [MLHR-684] - Invalidate dashboard state from previous versions
  • [MLHR-881] - ETL Web App - Packaging
  • [MLHR-882] - ETL Web App - Dashboard
  • [MLHR-885] - ETL Web App - Sample Data Generation
  • [MLHR-997] - ETL Web App - Historical Data from MongoDB
  • [MLHR-1014] - Remove config properties edit page, replace with config home
  • [MLHR-1052] - Bar Chart Widget
  • [MLHR-1053] - Widget Library - Demo Application

Story

  • [MLHR-703] - Logstream UI
  • [MLHR-709] - Widgets Library as Independent Project

Task

  • [MLHR-661] - Migrate JDBC adapters to use the new database adapter interface
  • [MLHR-866] - Annotate all stateless operators in Malhar as such
  • [MLHR-868] - Demo Guide for Map-Reduce Operator - LogsCountApplication
  • [MLHR-869] - Demo guide for Map-Reduce Operator - NewWordCountApplication.
  • [MLHR-870] - Demo guide for Map-Reduce Operator - InvertedIndexApplication.
  • [MLHR-922] - Machine data demo is using deprecated attributes. Gives warnings on launch
  • [MLHR-951] - Remove deprecated warnings from demos
  • [MLHR-954] - Need to debug older demos
  • [MLHR-961] - Develop a Goldengate input adapter
  • [MLHR-970] - Re-create the application that we build for iAd Poc in Malhar
  • [MLHR-1006] - In Logical View do not include the unifier stats
  • [MLHR-1007] - Simple user authentication/authorization support in UI
  • [MLHR-1018] - Need a Kafka-HBase app
  • [MLHR-1023] - Update installer text on welcome screen
  • [MLHR-1024] - Create issues summary page under configuration
  • [MLHR-1025] - Add progress status indicator for installer wizard
  • [MLHR-1026] - Reorder install wizard screens
  • [MLHR-1027] - Remove external upgrade link from licensing screen
  • [MLHR-1041] - ETL : Consolidate input operator properties in json format
  • [MLHR-1044] - ETL: Add input operator to ETL Application in ETL branch
  • [MLHR-1071] - Create stateless deduper - aka deduper which forgets its state upon failure or repartitioning

Version 0.9.4

Bug

  • [SPOI-455] - Cleanup maven repository workaround in install.sh
  • [SPOI-1636] - Update all node1 demo to 0.9.3 release
  • [SPOI-1774] - Thread local performance drop from 65M to 40 M tuplesProcessed/sec
  • [SPOI-1775] - MROperator demo applications fail when launched from Sandbox
  • [SPOI-1847] - Intermittent - WebSocket Publishing ignores endWindow
  • [SPOI-1864] - POC partitioned operators not getting correct initial state
  • [SPOI-1866] - Datatorrent applications not starting on CDH5 Vagrant cluster
  • [SPOI-1867] - Investigate InstallAnywhere use for DataTorrent installation
  • [SPOI-1882] - When not able to contact license agent, the application should not die
  • [SPOI-1883] - Stram crashes with Unknown-Container issue
  • [SPOI-1885] - Determine minimum amount of memory needed to run twitter app
  • [SPOI-1886] - Determine minimum amount of memory needed to run mobile app
  • [SPOI-1888] - Install license in the new console configuration
  • [SPOI-1889] - Licensing needs to support eval and free
  • [SPOI-1902] - Dynamic MxN partitioning does not handle scale down to single M instance
  • [SPOI-1903] - MiniClusterTests fails because ~/.dt/dt-site.xml dependency
  • [SPOI-1906] - history does not get flushed to the history file until the next command prompt
  • [SPOI-1917] - Licensing error
  • [SPOI-1922] - Loading license does not work
  • [SPOI-1923] - Close button does not work in several pop-up windows
  • [SPOI-1936] - No uninstall script available with the installer.
  • [SPOI-1953] - Installation video on our website should be refreshed to reflect the latest version(s)
  • [SPOI-1960] - application incorrectly marked as succeeded
  • [SPOI-1968] - 404 for logical plan url
  • [SPOI-1970] - Negative requested container count logged by Stram
  • [SPOI-1976] - Fix AdsDimensions in certification
  • [SPOI-1992] - the latency in the application overview freezes after a container gets killed.
  • [SPOI-1999] - Evaluate Yarn cluster issue
  • [SPOI-2002] - Gateway fails to load properties
  • [SPOI-2037] - Redirect to welcome page on first run / install
  • [SPOI-2039] - DTGateway logs to .dt/logs in service mode
  • [SPOI-2040] - Apache logs application under contrib folder fails to run
  • [SPOI-2047] - dtgateway service not starting when machine boots
  • [SPOI-2049] - Better error message on invalid hadoop directory
  • [SPOI-2051] - RPM install reports install script failure
  • [SPOI-2052] - DTGateway restart fails when running with -service
  • [SPOI-2055] - dtgateway service reports OK on startup failure
  • [SPOI-2056] - LicensingAgentProtocolHelper.getLicensingAgentProtocol gets stuck when YARN is not running
  • [SPOI-2057] - DTCLI is not working after intallation
  • [SPOI-2058] - launch-demos macro not available after installing the platform from self extracting intaller
  • [SPOI-2059] - Show understandable error message if the root hdfs directory could't be created
  • [SPOI-2060] - UI Shows Nan B in Allocated Memory
  • [SPOI-2062] - Gateway needs to check hadoop version
  • [SPOI-2064] - dtgateway-ctl stop doesn't work
  • [SPOI-2065] - Readme File is not udpated
  • [SPOI-2066] - installer not recognizing -l option
  • [SPOI-2070] - Installer: echo_success command doesn't work with Ubuntu
  • [SPOI-2073] - Invalid Ip Address in Installer UI
  • [SPOI-2074] - install script needs to check hadoop version
  • [SPOI-2075] - lauch-local-demos
  • [SPOI-2076] - Licensing agent RPC gives NPE
  • [SPOI-2077] - Installer: we need a separate page for hadoop installation directory
  • [SPOI-2078] - Change the certification and benchmarking script and code to use the new location of benchmarking apps
  • [SPOI-2083] - install script, when run by rpm, complains about invalid group dtadmin
  • [SPOI-2088] - Map Reduce demo applications still show the classnames when listed using launch-demos
  • [SPOI-2089] - demo applications displayed after running the launch-demos command should be in alphabetical order
  • [SPOI-2090] - Error while requesting evaluation license from Datatorrent.com
  • [SPOI-2094] - Installer throws failed message while stopping gateway
  • [SPOI-2097] - launch-demos macro not available after installing the platform from self extracting intaller
  • [SPOI-2098] - App names still have full classpath
  • [SPOI-2120] - Installer - Restart Modal is not closed after Restart Failed (Happened Once)
  • [SPOI-2132] - Ensure HDFS does not blow up with millions of files per sec
  • [SPOI-2133] - Delete old files to ensure NN does not crash
  • [SPOI-2134] - Send POC1 to customer
  • [SPOI-2143] - Spelling error and reference to $HOME/.datatorrent
  • [SPOI-2146] - Move Kafka benchmark apps to contrib folder
  • [SPOI-2148] - Installer - Disable Closing Modals on Click
  • [SPOI-2149] - Address the confusion around gateway address.
  • [SPOI-2151] - User is not able to change the defaultFS during installation
  • [SPOI-2153] - Cryptic error message when launching app on node0
  • [SPOI-2157] - Getting logical plan returns error when one of the getters is bad
  • [SPOI-2159] - gateway is polling resourcemanager for appinfo w/o subscriber
  • [SPOI-2163] - Change directory before DTGateway launch
  • [SPOI-2165] - Installer - add reload ability to System screen
  • [SPOI-2170] - DTGateway classpath is duplicated after restart
  • [SPOI-2171] - Remove reload button from System configuration screen
  • [SPOI-2172] - Installer may display invalid port after starting DTGateway
  • [SPOI-2173] - Installer base location change not working
  • [SPOI-2192] - CLI command for getting list of operator classes from a jar
  • [SPOI-2193] - CLI command for getting properties of a operator class from a jar
  • [SPOI-2206] - dag view does not get rendered property in Firefox
  • [SPOI-2227] - (Re)start license agent when license file is uploaded
  • [SPOI-2229] - container local operators not redeployed
  • [SPOI-2238] - Installer complains about sudo running as root
  • [SPOI-2241] - DAG Firefox 28 Support

Improvement

  • [SPOI-1311] - Review platform documentation
  • [SPOI-1567] - Certify against commercial Hadoop distributions
  • [SPOI-1769] - Trying to kill a non dt app can return a better message
  • [SPOI-1844] - Ship project-template with a log4j.properties file with debugging level set to INFO
  • [SPOI-1852] - WebSocket client recovery logging
  • [SPOI-1853] - Create WebSocket clients in containers on demand
  • [SPOI-1854] - Option to retrieve only running and recently-ended apps
  • [SPOI-1856] - Need something that is like LicensingAgentClient but not specific for stram.
  • [SPOI-1857] - Gateway to warn about available licensed memory being low
  • [SPOI-1858] - CLI to directly connect to license agent to get live license info
  • [SPOI-1859] - Gateway to directly connect to license agent to get live license info
  • [SPOI-1861] - Gateway command to restart itself
  • [SPOI-1869] - Add UI build script to dist build file
  • [SPOI-1874] - the first operator that stalls for more than specific period, take it out so as to unclog the processing
  • [SPOI-1899] - Add appmaster container to container list
  • [SPOI-1904] - Updates needed to the README file
  • [SPOI-1911] - Run certification as part of nighty build
  • [SPOI-1915] - Using $DT_HOME in README
  • [SPOI-1961] - Show the activation date of the license with list-license-agent command
  • [SPOI-1965] - The file demos.jar should be installed by default by the Installer
  • [SPOI-1978] - Manual eval request (by e-mail) - template
  • [SPOI-1980] - DT Version in license request and generated license
  • [SPOI-1981] - Approve / update license verification e-mail
  • [SPOI-1997] - Certify against commercial Hadoop distributions
  • [SPOI-2026] - Add support to LogicalPlan for testing dag serialization
  • [SPOI-2150] - Update Readme file for the local install
  • [SPOI-2155] - Installer - Validate Fields on blur event
  • [SPOI-2156] - Installer - Navigation Code Cleanup
  • [SPOI-2158] - Installer - CSS Classes
  • [SPOI-2160] - Installer font size
  • [SPOI-2166] - Configuration screen navigation panel
  • [SPOI-2174] - Notify user with installation location and version
  • [SPOI-2175] - Notify user about local DTGateway management during installation
  • [SPOI-2188] - Installer - Register Instructions

New Feature

  • [SPOI-328] - Add annotation to declare the operator stateless
  • [SPOI-393] - High Availability for STRAM
  • [SPOI-868] - Setting operator properties as different types
  • [SPOI-1182] - Add Key-based filter functionality to malhar library for Min, Max, SimpleMovingAverage, StandardDeviation like operators
  • [SPOI-1654] - Logstream - aggregate top hits and bytes for URL, geo DMA, IP, URL/status code, url
  • [SPOI-1747] - create a filter operator to output multiple records based on filter
  • [SPOI-1756] - configuration for input adaptor
  • [SPOI-1757] - configuration for filter operator
  • [SPOI-1758] - configuration for dimension operator
  • [SPOI-1759] - configuration for aggreagation operator
  • [SPOI-1760] - configuration for web socket output
  • [SPOI-1849] - add dt HDFS directory in configuration
  • [SPOI-1907] - Installer: HDFS directory creation attempt via Gateway (as part of config updates)
  • [SPOI-1909] - Port re-selection by Gateway if 9090, 9091, 9092, etc are taken
  • [SPOI-1913] - Automate verifing the app memory for the demos
  • [SPOI-1928] - gateway needs to be able to start with standalone hadoop jar (without hadoop installation)
  • [SPOI-1929] - /ws/v1/about to include java version and hadoop location
  • [SPOI-1930] - New installation script
  • [SPOI-1966] - dtcli should be enabled to list app names (if available) as opposed to app class path
  • [SPOI-1974] - Add throughput, totalTuplesProcessed and elapsed time to performance benchmarking
  • [SPOI-1975] - Display throughput, tuplesProcessed per sec and latency in a tabular format.
  • [SPOI-1987] - Copy License to Front-End Distribution
  • [SPOI-2003] - Verify and add all the demos except Machine data to app memory usage automation
  • [SPOI-2011] - Make a separate jar file for performance benchmarking demos
  • [SPOI-2013] - Support for doc link as an attribute for the application
  • [SPOI-2018] - Have a launch-performance command in dtcli
  • [SPOI-2021] - Rename all the apps under contrib to have meaningful names
  • [SPOI-2023] - Make a launch-contrib command available in stram cli
  • [SPOI-2027] - Packaging benchmarking demos
  • [SPOI-2042] - redirect user to welcome screen if dt.configStatus is not “complete”
  • [SPOI-2044] - set property dt.configStatus to “complete” when the user has completed the config wizard
  • [SPOI-2122] - Installer - Offline Email Template
  • [SPOI-2147] - Provide separate dt benchmarking package scripts to throughput and hdfs operators benchmarking
  • [SPOI-2195] - Gateway REST API to return operator classes in a jar given superclass (or not)
  • [SPOI-2196] - Gateway REST API to return properties of an operator in a jar
  • [SPOI-2200] - Installer - License Flow

Story

  • [SPOI-1608] - Platform Benchmarking (Platform1 and Platform2)

Task

  • [SPOI-722] - Document ads demo (add comments to codebase)
  • [SPOI-1403] - HDFS Operator Benchmark
  • [SPOI-1411] - Deprecate old Malhar webdemos once logstream is available
  • [SPOI-1513] - datatorrent.com webapp development - app testing
  • [SPOI-1610] - Develop benchmarking app for AdsDimension App (exactly once semantics) - Platform1
  • [SPOI-1612] - Benchmarking Ads Dimension demo app on Morado cluster (exactly once semantics) - Platform1
  • [SPOI-1618] - Benchmarking Machine Data app with Platform1
  • [SPOI-1694] - Document DT SandBox preparation
  • [SPOI-1730] - Default License in git needs to be replaced by the license cut by the real key
  • [SPOI-1732] - Create the real public/private key for licenses and store the private key in a safe place.
  • [SPOI-1745] - Document Demo Application Setup
  • [SPOI-1788] - CLI commands for licensing
  • [SPOI-1794] - Create license info as a string
  • [SPOI-1796] - Soft enforcement for normal paid app. 200% bump?
  • [SPOI-1848] - Gateway to support changing and getting config parameters
  • [SPOI-1851] - Document virtual cluster setup
  • [SPOI-1862] - Working on creating Wire Frames for the Installation of DT Platform
  • [SPOI-1863] - Make installer work w/o Maven
  • [SPOI-1865] - Allow user to configure application classpath
  • [SPOI-1868] - Support Book Keeping in the HDFSStorage
  • [SPOI-1870] - Validate dtcli generate-license-request
  • [SPOI-1872] - Modify generatelicense process
  • [SPOI-1873] - License process via console
  • [SPOI-1876] - Application Developer Guide Improvements
  • [SPOI-1877] - Download and build JDK Standrd Doclet Source as part of DT
  • [SPOI-1878] - DT Console Web UI Testing on Chrome
  • [SPOI-1879] - DT Console Web UI Testing for Demo Apps on Chrome
  • [SPOI-1884] - Operater developer guide review
  • [SPOI-1891] - Add allatori documentation
  • [SPOI-1892] - Add automatic build of front components to distribution
  • [SPOI-1893] - Quick Start Guide
  • [SPOI-1914] - cli get-app-info to include info from hadoop
  • [SPOI-1926] - Call license web service and return license file
  • [SPOI-1927] - Returns license request blob for UI to assemble mailto link
  • [SPOI-1939] - Twitter Top Counter Demo Applications Guide
  • [SPOI-1940] - RPM packaging for installer
  • [SPOI-1941] - Include demo UI into installer
  • [SPOI-1942] - Remove Allatori code expiration for GA
  • [SPOI-1943] - include more info in the license request
  • [SPOI-1945] - Add REST call to gateway to post license file
  • [SPOI-1946] - virtual cluster configuration changes
  • [SPOI-1947] - Create license@datatorrent.com
  • [SPOI-1949] - Java application (with main method) that returns information given a license request blob
  • [SPOI-1956] - License generation key pair expiration / private key protection
  • [SPOI-1963] - Evaluate Doclava Doclet from Google
  • [SPOI-1982] - E-mail verification success web page
  • [SPOI-1988] - Review Quick Start Guide
  • [SPOI-1989] - AdsDimension - Demo Applications Guide
  • [SPOI-1990] - Twitter Rolling Top Words Counter - Demo Applications Guide
  • [SPOI-2004] - Installer testing for GA
  • [SPOI-2006] - Grant Google Analytics Access To Following People
  • [SPOI-2010] - Configure the apps to use minimum memory as verified by app-memory-usage-automation.
  • [SPOI-2015] - Get Machine data into contrib.jar
  • [SPOI-2016] - Fraud Detection in contrib.jar
  • [SPOI-2017] - Quick Start Guide version 2
  • [SPOI-2019] - List NxN performance apps (different event size vs different stream locality
  • [SPOI-2025] - Getting Start Guide - Launch this copy.
  • [SPOI-2032] - Certify Cloudera CDH 5.0
  • [SPOI-2048] - Uninstall script
  • [SPOI-2050] - Start gateway as service flag
  • [SPOI-2069] - Test Installer
  • [SPOI-2091] - Update installation license agreement
  • [SPOI-2093] - Verify demo UI is bundled with installer
  • [SPOI-2100] - For terms of service box, change “continue” to “accept and continue”
  • [SPOI-2106] - Change the message on 1 yr registration
  • [SPOI-2107] - Change the message on 1 yr registration
  • [SPOI-2108] - Put timeglass and “loading” or spinning.... while Hadoop system properties are being loaded
  • [SPOI-2109] - Gateway down creates bad error message
  • [SPOI-2110] - Remove errors popping on the right hand of console
  • [SPOI-2111] - If gateway outage is discovered add a message to get them back
  • [SPOI-2112] - Change the message on Hadoop screen
  • [SPOI-2114] - Error if HDFS does not exist
  • [SPOI-2115] - Create a list of issues summary screen
  • [SPOI-2116] - Show more instructions on the completed screen
  • [SPOI-2131] - Ingestion POC
  • [SPOI-2136] - Do cartesian products for key, val pair
  • [SPOI-2142] - Allow customization of cartesian product
  • [SPOI-2161] - prereq message on welcome screen
  • [SPOI-2162] - DFS error message
  • [SPOI-2164] - DFS location validation
  • [SPOI-2167] - Evaluate errors in Cloudera certification
  • [SPOI-2223] - Test if all the applications with reduced container memory sizes run in sandbox 0.9.4.
  • [SPOI-2230] - Uninstaller for RPM
  • [SPOI-2231] - Provide Environment with Running Demos
  • [SPOI-2240] - Set Up DataTorrent Demos on Dev Environment

Sub-task

  • [SPOI-1682] - Too many mbassy threads!!!
  • [SPOI-1718] - Update HA Documentation
  • [SPOI-1729] - Restore operator recovery checkpoints in AM recovery
  • [SPOI-1786] - Users should be able to generate license at datatorrent.com
  • [SPOI-1931] - Installer - determine OS type and version
  • [SPOI-1932] - installer with sudo/root user creation, service installs
  • [SPOI-1933] - service wrappers for DTGateway
  • [SPOI-1934] - HDFS directory creation during install
  • [SPOI-1935] - Search for hadoop binaries standard paths
  • [SPOI-1958] - Create HDFS Word input operator
  • [SPOI-1962] - Add test to jekins nightly build
  • [SPOI-1971] - Verify launch script for other apps from demo
  • [SPOI-1972] - When some app fail, the main monitor should still keep looking at the other apps
  • [SPOI-2028] - Provide shell script ‘benchmark-throughput’ to produce a single summary table
  • [SPOI-2030] - Provide a list of individual demos used for benchmarking through dtcli benchmarking, so that the user can launch the demos individually.
  • [SPOI-2031] - Package benchmarking suite into the installer and sandbox
  • [SPOI-2046] - Update licenses location in installation script
  • [SPOI-2053] - Certify CDH5.0 as part of Cloudera certification for inclusion in their process
  • [SPOI-2054] - Certify installer on HW
  • [SPOI-2079] - Run certification on bin install
  • [SPOI-2085] - Change the certification resource xml files to contain the certification type.

Bug

  • [MLHR-729] - Columns in table wrap in Firefox
  • [MLHR-730] - Dev server does not escape double-quotes in error message
  • [MLHR-732] - Links Outline in Firefox
  • [MLHR-754] - JarList and DepJarList headers point to nonexistent text items
  • [MLHR-767] - “FINISHING” has no icon in status
  • [MLHR-785] - NaN in file size column of jar list
  • [MLHR-809] - Inconsistent landing page across browsers
  • [MLHR-814] - Selecting operators in the logical operators widget does not activate any actions
  • [MLHR-816] - Names of partitions in the partitions widget should not be shown as links
  • [MLHR-817] - Clickin on ‘outputPort’ leads to PageNotFound error
  • [MLHR-824] - Don't show license agent detail when license agent is not running
  • [MLHR-826] - Dep Jars fail to load in specify dep jars modal
  • [MLHR-830] - Optimize RandomWordInput operator used in perfromance benhcmarking to use local final variables to improve performance.
  • [MLHR-835] - Fix hard-coded file path in com.datatorrent.demos.wordcount.WordCountInputOperator
  • [MLHR-836] - Need AbstractHDFSOutputAdapter
  • [MLHR-863] - Add license headers to AbstractHdfsOutputOperator
  • [MLHR-879] - Installer - Issues Management
  • [MLHR-887] - Input operator that tails a growing log file in a directory
  • [MLHR-899] - Give default name for all demo applications
  • [MLHR-908] - Mark installation complete with dt.configStatus property
  • [MLHR-911] - UI Shows Nan B in Allocated Memory
  • [MLHR-912] - Installer - System Section - Show Field Specific Errors
  • [MLHR-913] - Installer - System Section - Server-Side Error Messages
  • [MLHR-918] - Yahoo finance with Alerts : Modify to accept multiple ticker symbols and remove hard-coded values.
  • [MLHR-920] - Give desciptive names to benchmarking apps
  • [MLHR-924] - Tail Operator should take care of the truncation of file
  • [MLHR-930] - UI container list should not show time for last heartbeat for if the value is “-1”
  • [MLHR-938] - gateway address property change
  • [MLHR-939] - Put GatewayRestart into “actions” hash in settings.js
  • [MLHR-940] - Delete the space in the name
  • [MLHR-943] - Change the method name from isConnected to Connected in the db api
  • [MLHR-991] - DAG Stream Locality

Improvement

  • [MLHR-731] - Compile LESS on the fly in dev environment
  • [MLHR-735] - Remove unused bundling make commands, update README
  • [MLHR-745] - Add icon to left widget manager drawer
  • [MLHR-746] - Use “cores” not % cpu for cluster metric total cpu usage
  • [MLHR-747] - Remove “#” for “number of container”, et al labels
  • [MLHR-748] - Only load all applications from RM on demand
  • [MLHR-749] - Improve notification history pane
  • [MLHR-750] - Normalize labels for everything in console
  • [MLHR-751] - Add lock where close dashboard icon would be for default dashboard
  • [MLHR-753] - Remove avg app age in cluster metrics
  • [MLHR-755] - Change “max alloc mem” to “peak alloc mem” in cluster metrics
  • [MLHR-756] - Add memory levels to tooltip of license mem gauge in top right
  • [MLHR-768] - Remove beefy from npm shrinkwrap
  • [MLHR-770] - Clean up BaseUtil, BaseModel, BaseCollection, add tests
  • [MLHR-781] - shorten link to log file in container page
  • [MLHR-798] - Add “config.adsdimensions.redis.dbindex” configuration in webdemo
  • [MLHR-873] - Add equals and hashcode to JdbcOperatorBase
  • [MLHR-931] - ETL: Create a converter api and provide an implementation for Json Object to flat map conversion
  • [MLHR-935] - Have “silentErrors” option for models and collections
  • [MLHR-937] - ETL: Create a unifier for DimensionComputation operator
  • [MLHR-947] - Improve overall look and feel of install wizard
  • [MLHR-948] - Remove mock issues from SummaryView in installer wizard
  • [MLHR-993] - Demo UI - Default Applications Names for Application Discovery

New Feature

  • [MLHR-688] - Discard Real-Time Updates When Page is not Active
  • [MLHR-757] - Dashboard - Save Widget Width to Local Storage
  • [MLHR-759] - Dashboard - Widget Definitions Collection
  • [MLHR-763] - Line Chart Widget - hAxis options
  • [MLHR-764] - Dashboard App - Meteor Data Source
  • [MLHR-765] - Shutdown Container Interface
  • [MLHR-773] - Configuration wizard page
  • [MLHR-788] - Web Demos - Redis and MongoDB Config
  • [MLHR-789] - Web Demos - Fraud Demo MongoDB Database Name
  • [MLHR-790] - Web Demos - Single Configuration File
  • [MLHR-791] - Web Demos - Start Script
  • [MLHR-792] - Web Demos - Single Page App
  • [MLHR-793] - Web Demos - JS/CSS Bundle
  • [MLHR-794] - Web Demos - WebSocket Pub/Sub
  • [MLHR-795] - Web Demos - Resources Clean Up on Scope Destroy
  • [MLHR-796] - Web Demos - Distribution Files
  • [MLHR-797] - Web Demos - Running Instructions
  • [MLHR-804] - Config Page (Manage Properties)
  • [MLHR-805] - Web Demos - Distribution Package Instructions
  • [MLHR-806] - Installer - License Requests (REST API Calls)
  • [MLHR-807] - Installer - License Text
  • [MLHR-818] - Installer - License Section
  • [MLHR-819] - Installer - System Properties Section
  • [MLHR-820] - Console - Node.js Dev Mode
  • [MLHR-821] - Web Demos - Distribution Package Launch Script
  • [MLHR-831] - Installer - License Flow
  • [MLHR-834] - Installer - License - Registration
  • [MLHR-858] - Installer - Gateway Restart
  • [MLHR-861] - Installer - System Properties - IP List
  • [MLHR-864] - Installer - Handling Hadoop Not Found
  • [MLHR-903] - Installer - WebSocket DataSource Disconnect
  • [MLHR-909] - Installer - Restart Confirmation
  • [MLHR-914] - Installer - Error Messages
  • [MLHR-919] - License Bar
  • [MLHR-923] - Installer - Properties Update “Loading” Indicator
  • [MLHR-925] - Installer Update
  • [MLHR-934] - Allow overrides to $.().modal(options) for Modals

Story

  • [MLHR-705] - Node.js Pub/Sub Service
  • [MLHR-708] - Evaluate Node.js Pub/Sub Services
  • [MLHR-710] - Pie Chart Data Model
  • [MLHR-712] - Dashboard App - Historical Data Support
  • [MLHR-714] - Dashboard App - MongoDB Integration
  • [MLHR-715] - Create Dashboard from Running App - Widgets Auto Discovery
  • [MLHR-736] - Console Firefox Support
  • [MLHR-737] - Console Safari Support
  • [MLHR-801] - Installer/Config UI
  • [MLHR-802] - Web Demos - Distribution Package
  • [MLHR-803] - Installer (Wizard) Page

Task

  • [MLHR-349] - Add build script to Malhar/front
  • [MLHR-402] - Logstream - aggregate top hits and bytes for URL, geo DMA, IP, URL/status code, url
  • [MLHR-651] - Use compatible version of jersey/jackson/jetty in Malhar
  • [MLHR-659] - Migrate MongoDB adapters to use the new database adapter interface
  • [MLHR-741] - Web Apps (Demos) Firefox Support
  • [MLHR-744] - Web Apps (Demos) Safari Support
  • [MLHR-760] - Dashboard App - Meteor Integration
  • [MLHR-761] - Dashboard App - Derby.js Integration
  • [MLHR-762] - create install script for ui
  • [MLHR-839] - Review guide on MachineData app
  • [MLHR-842] - Demo guide for Pi Application
  • [MLHR-843] - Demo guide for Twitter Top URL Counter demo application
  • [MLHR-851] - Demo guide for Fraud detection demo application
  • [MLHR-853] - Demo guide for Mobile demo application
  • [MLHR-854] - Demo Guide for Word-Count Application
  • [MLHR-855] - Demo guide for Pi Calculator application
  • [MLHR-856] - Demo guide for Twitter Rolling Top Words Application
  • [MLHR-859] - upgrade kryo to 2.23
  • [MLHR-871] - Demo guide for Twitter Top URL Counter - Launch This Copy.
  • [MLHR-872] - Demo Guide for Word-Count Application - Launch this copy.
  • [MLHR-875] - Demo guide for Yahoo finance application
  • [MLHR-876] - Demo guide for Yahoo finance alerting application
  • [MLHR-877] - Demo guide for Yahoo finance application with Derby SQL
  • [MLHR-892] - ETL logstream application - study the log stream application
  • [MLHR-893] - ETL- Use the generic dimension operator that was created for a POC in Log stream
  • [MLHR-900] - ETL- Operators used by logstream application need to be generic and moved to library
  • [MLHR-904] - Fix the nightly and trigger builds broken due to removal of api.codec and api.util
  • [MLHR-905] - Dedup: Make deduper and bucket manager part of malhar library
  • [MLHR-910] - Demo guide for Twitter Rolling Top Words Application - Launch This Copy
  • [MLHR-915] - CLONE - Demo guide for Pi Application - Launch this copy
  • [MLHR-916] - CLONE - Demo guide for Pi Calculator application - Launch this copy
  • [MLHR-917] - CLONE - Demo guide for Mobile demo application - Launch this copy
  • [MLHR-936] - Create new Redis Store using Lettuce redis client
  • [MLHR-949] - Add confirmation to DTGateway restart button in System Properties
  • [MLHR-962] - ETL : Create a sifter operator
  • [MLHR-980] - CLONE - Demo guide for Yahoo finance alerting application - launch this copy
  • [MLHR-981] - CLONE - Demo guide for Yahoo finance application - launch this copy
  • [MLHR-982] - CLONE - Demo guide for Yahoo finance application with Derby SQL - launch this copy

Sub-task

  • [MLHR-678] - Time Series Forecasting with Simple Linear Regression
  • [MLHR-718] - Time Series Forecasting using Simple/Single Exponential Smoothing
  • [MLHR-726] - Time Series Forecasting Operator using Holt's Linear Trend Model
  • [MLHR-727] - Develop application for a telecom related use case for time series forecasting with Simple Linear Regression and CMA smoothing
  • [MLHR-932] - Create Centered Moving Average Smoothing Operator

Version 0.9.3

New Feature

  • [SPOI-261] - Design a general purpose read from stream and write to cassandra
  • [SPOI-400] - Each streaming application should license check
  • [SPOI-1622] - Input operator - XML parser
  • [SPOI-1647] - LogStream UI
  • [SPOI-1770] - Gateway should expose list of available topics
  • [SPOI-1778] - Open readme on sandbox startup
  • [SPOI-1804] - Start license app on launch app if not running
  • [SPOI-1805] - command to show license file info in cli
  • [SPOI-1812] - Create REST call for specific license agent, given a license id
  • [SPOI-1823] - Gateway REST API - Get Running Applications List
  • [SPOI-1829] - Semantic URLs for Web Apps

Improvement

  • [SPOI-1202] - Provide a way to check whether an operator is partitioned
  • [SPOI-1783] - Add allocatedMB to main application list
  • [SPOI-1789] - Change frequency of heartbeats to license app
  • [SPOI-1795] - License file to clearly state hard enforcement or soft enforcement
  • [SPOI-1816] - support simple variable substitution in the cli
  • [SPOI-1835] - support gateway status command

Story

  • [SPOI-1320] - Support MQTT protocol
  • [SPOI-1542] - Input operator - Directory Scan

Bug

  • [SPOI-1696] - Make de-duper dynamically partitionable
  • [SPOI-1704] - Stram enforcement to lock physical plan changes when license memory limit is reached
  • [SPOI-1706] - Design a enforcement format for the license policy
  • [SPOI-1711] - Encryption/obfuscation of communication between stram and license agent
  • [SPOI-1779] - Update sandbox documentation terminology
  • [SPOI-1802] - Provide total license memory via stram web services
  • [SPOI-1807] - Unknown outage on Machine data demo
  • [SPOI-1809] - Suppress expected error message in dtgateway-ctl
  • [SPOI-1810] - sample-stram-site.xml generates warnings
  • [SPOI-1813] - Add SNAPSHOT repository to install pom
  • [SPOI-1815] - Make stram memory reporting to license manager asynchronous
  • [SPOI-1818] - Chance “className” in license file to id
  • [SPOI-1820] - dtcli script doesn't exit when maven command fails
  • [SPOI-1826] - Update documentation title
  • [SPOI-1827] - Install script errors
  • [SPOI-1831] - CLI warning about trouble with license manager
  • [SPOI-1833] - Use encrypted byte arrays for RPC wire protocol for licensing
  • [SPOI-1840] - Change default license memory limit to 25GB
  • [SPOI-1841] - Make stram memory enforcement tolerances property settings for the enforcer
  • [SPOI-1842] - Investigate the possibility of engine obfuscation jar not containing any references to license package path
  • [SPOI-1880] - GET nonexistent container returns 500 error

Task

  • [SPOI-1375] - All sandbox apps must work in 8G VM. Need to test each
  • [SPOI-1467] - DB lookup for Cassandra
  • [SPOI-1507] - datatorrent.com webapp development - pilot test of Angular and WP integration
  • [SPOI-1509] - datatorrent.com webapp development - db design
  • [SPOI-1511] - datatorrent.com webapp development - app design
  • [SPOI-1512] - datatorrent.com webapp development - app dev
  • [SPOI-1515] - datatorrent.com webapp deveopment - integrate standalone app with cms
  • [SPOI-1516] - datatorrent.com webapp development - add GA info during registration
  • [SPOI-1517] - datatorrent.com webapp development - background jobs
  • [SPOI-1617] - Benchmarking Performance app with Platform2
  • [SPOI-1641] - Benchmarking Ads Dimension App - Platform2
  • [SPOI-1715] - Show remainingLicensedMB and allocatedMB in UI for each application
  • [SPOI-1763] - Provide support for Accumulo NoSQL db
  • [SPOI-1780] - Sandbox - activate license automatically
  • [SPOI-1781] - Sandbox - increase memory to 8GB
  • [SPOI-1782] - License App should use much less memory (256MB or less?)
  • [SPOI-1787] - Add license instructions to README
  • [SPOI-1790] - Ensure update to license app on any resource change by StrAM
  • [SPOI-1791] - Hard enforcement for free license (6GB), and eval license
  • [SPOI-1793] - Hide sub-license and make license object behave as “what is license data right now?”
  • [SPOI-1798] - Date format change in license file
  • [SPOI-1799] - Change name “Sublicense” to “Section” or “LicenseSection”
  • [SPOI-1800] - Webservice specs for Gateway for license info
  • [SPOI-1825] - Update end user documentation
  • [SPOI-1832] - Support CDH default log4j setup

Sub-task

  • [SPOI-1451] - Show critical path
  • [SPOI-1721] - Augment the Partitionable interface to inform of all the partitions which actually were deployed
  • [SPOI-1733] - Container heartbeat RPC failover

New Feature

  • [MLHR-5] - UI component for license information
  • [MLHR-6] - Google Line Chart Widget
  • [MLHR-7] - Gauge Widget
  • [MLHR-8] - Top N Widget
  • [MLHR-9] - Compile Widget from HTML Template
  • [MLHR-10] - Make widgets resizable and renameable in ui-builder
  • [MLHR-11] - Dashboard Component Grunt Tasks
  • [MLHR-653] - Dynamically Connect Widgets to WebSocket Topics
  • [MLHR-655] - Create serializing mechanism for instantiated widgets and dashboard(s)
  • [MLHR-656] - Add/Compile Widgets from templateUrl
  • [MLHR-664] - Support MQTT protocol
  • [MLHR-668] - Set up widget configure dialog
  • [MLHR-669] - Visual Data Demo App
  • [MLHR-671] - Add allocatedMB column in main application list
  • [MLHR-673] - Dashboard App - Notification Service
  • [MLHR-674] - Explicit Saving/Loading of Dash configurations in ui builder
  • [MLHR-687] - Dashboard App - Filter WebSocket Topics
  • [MLHR-689] - Dashboard App - Widget Options Modal
  • [MLHR-690] - Dashboard App - Widgets Schema
  • [MLHR-691] - WebSocket Topics Debugger Widget
  • [MLHR-692] - JSON Widget
  • [MLHR-693] - Progressbar Widget
  • [MLHR-695] - Pie Chart Widget
  • [MLHR-696] - Dashboard App - Development/Production Scripts
  • [MLHR-697] - Dashboard App - Node.js Configuration
  • [MLHR-698] - Dashboard App - WebSocket/REST API Configuration

Improvement

  • [MLHR-686] - Deglobalize the visibly component

Bug

  • [MLHR-4] - Use new livechart module for OpChart Widget
  • [MLHR-13] - Console status column display
  • [MLHR-648] - Update Issues section in README files
  • [MLHR-650] - Changing metrics on logical dag fails
  • [MLHR-654] - Some widgets' height changes when changing width
  • [MLHR-667] - Add UI version to console
  • [MLHR-670] - Memory leak in console
  • [MLHR-677] - Widgets Data Models
  • [MLHR-680] - Update license information dialog for new REST call info
  • [MLHR-725] - WindowId Formatter
  • [MLHR-739] - Stream Locality Toggle fails for DAG view

Story

  • [MLHR-1] - Reusable Dashboard Component with AngularJS
  • [MLHR-2] - Dashboard Widgets
  • [MLHR-3] - Dashboard App

Task

  • [MLHR-321] - Directory Scan operator
  • [MLHR-452] - Create a De-duplication operator
  • [MLHR-603] - Supports upload of dependency jars
  • [MLHR-638] - Test streaming application for dynamic partition
  • [MLHR-645] - More fields in drop down in logicalDAG widget
  • [MLHR-646] - Document issue tracking location in README
  • [MLHR-652] - Parallel Simple Linear Regression
  • [MLHR-657] - Migrate memcache adapters to use the new database adapter interface
  • [MLHR-662] - Migrate Redis adapters to use the new database adapter interface
  • [MLHR-663] - Design new DB adapters interface
  • [MLHR-666] - DB lookup for Cassandra

Version 0.9.2

Bug

  • [SPOI-1327] - AtLeastOnceTest.testInlineOperatorsRecovery intermittent failure
  • [SPOI-1342] - DTCli should check license and relay the information with each application launched
  • [SPOI-1383] - Last window id and recovery window id do not update on 0.9
  • [SPOI-1439] - Gateway should be secured
  • [SPOI-1445] - Add version detection for Gateway
  • [SPOI-1456] - Free Memory in container widget changes too rapidly
  • [SPOI-1540] - Specification of license handlers and enforcers in the license file.
  • [SPOI-1632] - jar upload fails
  • [SPOI-1634] - Uptime at 1Billion events/s (Machine data)
  • [SPOI-1635] - Update node1 with latest machine data demo
  • [SPOI-1676] - Incremental Obfuscation of dt-flume directory
  • [SPOI-1677] - Supports uploading of dependency jars
  • [SPOI-1678] - When loading jars, make sure they are in their separate space so they don't conflict with gateway, cli and other jars
  • [SPOI-1679] - When uploading jar and when dependencies are not met, allow the upload with a message about dependencies
  • [SPOI-1680] - gateway throws errors when retrieving web service info from stram
  • [SPOI-1687] - Support launching jar and showing logical plan from HDFS
  • [SPOI-1688] - Map Reduce Monitor Does Not Publish WebSocket Data
  • [SPOI-1697] - Update demo configuration on node2
  • [SPOI-1703] - Update auto provisioning with DataTorrent 0.9.1 and GCE GA
  • [SPOI-1707] - License agent should handle license expiry
  • [SPOI-1708] - Stram should store license expiry
  • [SPOI-1709] - Show license object information in gateway
  • [SPOI-1710] - License cutting utility
  • [SPOI-1712] - Gateway to gracefully handle stram being a newer version than itself
  • [SPOI-1714] - Dynamic partition stop working if you start from only 1 partition
  • [SPOI-1727] - ApplicationInfoAutoPublisher unit test error
  • [SPOI-1728] - StramEvent exception prevents package name obfuscation
  • [SPOI-1739] - recordingStartTime of operator stats is showing -1 from time to time
  • [SPOI-1743] - Tuple recording on port is not showing up in web services
  • [SPOI-1744] - Recording says ended even if the recording is still going on

Improvement

  • [SPOI-1098] - event recorder logging improvements
  • [SPOI-1370] - Make the partition logic available to the end-users
  • [SPOI-1448] - DAG Visualization - Stream Types
  • [SPOI-1603] - BufferServerStatsCollection - dont check against bufferserverpublisher and subscriber
  • [SPOI-1613] - Update the User Interface guide to reflect latest version (0.9.1)

New Feature

  • [SPOI-165] - Parent jira for authentication
  • [SPOI-170] - Stream should authenticate before allowing an operator to connect
  • [SPOI-258] - Develop Flume Sink and corresponding DT input adapter
  • [SPOI-327] - Parent jira for Security
  • [SPOI-401] - Licensing alert mechanisms
  • [SPOI-411] - Ability to modify (add, upgrade, downgrade) license while the app is running
  • [SPOI-436] - Provide Web Service for obtaining license information (Usage limits, etc)
  • [SPOI-729] - Include license data in DT phone home
  • [SPOI-872] - Logical View of Running Application
  • [SPOI-975] - Support DataLocal Functionality
  • [SPOI-1406] - Add log file path and/or URL to each container info map
  • [SPOI-1621] - Input operator - CDR parser using CSV
  • [SPOI-1699] - Add locality (and maybe id?) to physical streams in REST calls

Task

  • [SPOI-1689] - Map Reduce Monitor Web App
  • [SPOI-739] - Hadoop 2.2 certification
  • [SPOI-763] - Competition study
  • [SPOI-1140] - Annotate dag visualization with stream throughout and other data
  • [SPOI-1246] - Support versioning for Gateway to STRAM communications
  • [SPOI-1253] - Create DataTorrent Application which provides licensing server functionality
  • [SPOI-1389] - ContainerList view should show the log file name (stderr, stdout) in the info widget
  • [SPOI-1405] - Design macros for node0 and node1
  • [SPOI-1609] - Competitive analysis - DT (Platform1 and Platform2)
  • [SPOI-1611] - Benchmarking Ads Dimension on Morado cluster (at-least-once semantics) - Platform1
  • [SPOI-1616] - Benchmarking Performance app with Platform1
  • [SPOI-1670] - Ensure that Dedup operator is fault tolerant
  • [SPOI-1673] - use public/private key encryption for dt phone home
  • [SPOI-1675] - Map Reduce Jobs
  • [SPOI-1686] - Launch separate process when loading classes from application jars
  • [SPOI-1722] - Create a utility to create default license
  • [SPOI-1724] - Create a command line utility to generate customer license
  • [SPOI-1736] - CLI warning on License Violation
  • [SPOI-1742] - Update end-user documentation

Sub-task

  • [SPOI-919] - Certify secure mode with Hadoop 2.2.0
  • [SPOI-966] - Create a licensing agent application
  • [SPOI-1413] - Flume sink part
  • [SPOI-1414] - DT Input Adapter for Flume
  • [SPOI-1475] - Augment Kafka operator to dynamically adapt to load and broker/partition changes
  • [SPOI-1538] - Develop Ads Dimension on Morado cluster (at-least-once semantics) Platform1
  • [SPOI-1713] - Secure communication between gateway and stram
  • [SPOI-1720] - Ensure that the Partionable interface and StatsListener interface callbacks are made from the same thread
  • [SPOI-1723] - The default license generation should be integrated with build
  • [SPOI-1731] - Sync execution layer deployment state after recovery

GitHub - DataTorrent/Malhar

  • [616] - fix #615 Update Web Apps Instructions
  • [615] - Update Web Apps Instructions
  • [614] - corrected typo
  • [613] - Fixes #599. Upload and specify dependency jars
  • [612] - fixes #597
  • [611] - fixes #610
  • [610] - Telecom tests failing
  • [609] - Github 597
  • [608] - #fix 607 Machine Data Demo Day Format
  • [607] - Machine Data Demo Day Format
  • [606] - fixes #457 added xml parse operator to parse and pick values from xml nodes and att...
  • [605] - added the history for hadoop 1.x
  • [603] - Map Reduce Monitor - Elapsed Time
  • [602] - Map Reduce Monitor - Elapsed Time
  • [601] - Map Reduce Monitor - Bootstrap JS, Server Errors Notification, Header Alignment
  • [599] - Provide UI for uploading and specifying dependency jars
  • [598] - Map Reduce Monitor - Server Errors Notification
  • [597] - Map Reduce Monitor App - CPU/Memory History
  • [595] - Map Reduce Monitor (History Charts, Animations, Readme, AngularJS Upgrade)
  • [594] - Map Reduce Monitor - Production Files (dist)
  • [593] - Map Reduce Monitor - AngularUI Bootstrap Progressbar issue with ngAnimate
  • [592] - Map Reduce Monitor Update (Readme, App List, History Charts)
  • [591] - Map Reduce Monitor - Map/Reduce History Charts
  • [590] - Fixes #401, adds zoom to physical DAG
  • [589] - Map Reduce Monitor - App List Columns
  • [586] - Map Reduce Monitor - App Id AngularJS Filter
  • [585] - Fixes #569, Cosmetic changes
  • [584] - Map Reduce Monitor - Show Active Job First
  • [581] - Map Reduce Monitor Update (Loading Indicator, Animations, Delayed Promise)
  • [580] - Map Reduce Monitor - AngularJS Animations
  • [579] - Map Reduce Monitor - Upgrade to AngularJS 1.2.6
  • [578] - Map Reduce Monitor - App List Loading Indicator
  • [577] - fixes #553, fixes #575
  • [576] - Map Reduce Monitor - AngularJS Delayed $q Promise
  • [575] - Map Reduce Monitor App - Send Job Stats immediately on Subscribe Request
  • [573] - Map Reduce Monitor Update (App List Grid, Progress Bars, Job Removal)
  • [572] - fixes #570
  • [571] - Map Reduce Monitor - Remove Job on WebSocket Message
  • [570] - clipPath issue when multiple charts on the same page
  • [569] - Various cosmetic updates for console
  • [568] - fixes #542, tooltip no longer obstructed by graph lines
  • [567] - Bird's Eye View for Physical DAG view
  • [566] - fixes #544, windowIds can now handle initial value of -1 or 0
  • [565] - Map Reduce Monitor - Job Selection
  • [564] - fixes #357, added logical operator page
  • [563] - Map Reduce Monitor - App List ng-grid
  • [561] - Map Reduce Monitor - App List Table Filter
  • [560] - Fixed exception with KryoSerializableStreamCodec #559
  • [558] - CDR simulator #524
  • [557] - Github 525
  • [556] - set the name of the io threads created by ning asynchttpclient
  • [554] - Squashed commit of the following:
  • [553] - Map Reduce Monitor App - Store Map/Reduce Progress History
  • [551] - fixes #550
  • [550] - Map Reduce Monitor App - App Should Broadcast Special Message on Unsubscription
  • [549] - Map Reduce Monitor - Stop Updates after Job Unsubscribe
  • [548] - Map Reduce Monitor - Found Job Notification
  • [547] - Fix Github #545
  • [544] - Console does not handle initial windowId
  • [542] - Tooltip for line graphs show up behind graphs after turning series on and off
  • [541] - fixes #535
  • [539] - Map Reduce Monitor - Merge Progress Bars with Progress Table
  • [538] - Map Reduce Monitor - Combine Map/Reduce Counters
  • [536] - Map Reduce Monitor - App List Running Jobs Progress Bar
  • [535] - Console breaks when switching to other page
  • [534] - Fixes #510, Unsubscribe Logical Operators when not used by any widget
  • [533] - Fixes #521, Refactored WindowId usage
  • [532] - Map Reduce Monitor - Counters
  • [530] - Map Reduce Monitor - App List Sort
  • [526] - Map Reduce Monitor - Counters
  • [525] - CDR processing DAG prototype
  • [523] - Github 512
  • [521] - Normalize all WindowId objects by overriding “set” method of appropriate models
  • [519] - Map Reduce Monitor - Header Alignment on Resize
  • [518] - Map Reduce Web App
  • [517] - Map Reduce Monitor - License Headers
  • [516] - Map Reduce Monitor - Map Reduce Jobs List
  • [515] - Map Reduce Monitor - AngularJS Modules Definition
  • [514] - Map Reduce Monitor - Readme (Deployment and Running Instructions)
  • [513] - Map Reduce Monitor - Job Query Loading Indicator
  • [512] - Support Normalization Operator
  • [511] - Map Reduce Monitor - AngularJS Settings Provider
  • [510] - Unsubscribe logicalOperators on InstancePage when not in use by widget
  • [509] - Fixes #505. Also removes one more instance of free memory metric for containers
  • [507] - Map Reduce Monitor - Single Config (Server and Client)
  • [505] - Add processed and emitted metrics to container overview widget
  • [504] - fixes #356, container log url now available in container info widget
  • [503] - Map Reduce Monitor - Active Job Highlight
  • [502] - Map Reduce Monitor - AngularJS Parent Scope Event Propagation
  • [501] - Fixes #364, removed free memory from container metrics
  • [500] - Map Reduce Monitor - Mock Server
  • [499] - Adding support for R. Basic operations - min, max and std deviation support added. Also adding support to run R scripts.
  • [498] - Rsupport pull
  • [496] - Map Reduce Monitor - Progress Line Chart
  • [495] - Map Reduce Monitor - Running MAPREDUCE Applications Discovery
  • [494] - fixes #420, can now set explicit height for widgets
  • [493] - fix #488 added delay before reconnection
  • [492] - CPU/RAM Metrics for Map Reduce Jobs
  • [491] - fix #488 added delay before reconnection
  • [489] - Map Reduce Monitor - Job Controller
  • [488] - WebSocketOutputOperator should wait a specified number of seconds before reconnection
  • [487] - Using uniform naming convention for applications. Fixed incorrect application names. Fixes #486.
  • [486] - Application names are not uniform
  • [485] - CPU/RAM Metrics for Map Reduce Jobs (Map Reduce Monitor App)
  • [484] - Map Reduce Monitor - AngularJS UI-Router Nested Views
  • [483] - Enhance the AbstractSlidingWindow #480, Add a SortedSlidingWindow operator #423
  • [482] - fixes #411. bundling on server.js, monkeypatching fs to avoid EMFILE
  • [479] - fix #443 reconnection when the connection is dropped
  • [478] - fix #443 Handles reconnection when the connection is dropped
  • [477] - Improvements to LogicalDagWidget. Fixes #399, #473, #475, #476
  • [476] - Logical DAG Widget: Limit scroll scale extent
  • [475] - Logical DAG Widget: add ability to reset initial dag view
  • [474] - Map Reduce Monitor - AngularJS UI-Router
  • [473] - Logical DAG Widget: only zoom when alt/option is held down
  • [472] - Map Reduce Monitor App - App Does not Publish Completed Maps
  • [471] - Map Redice Monitor - Reduce Progress Grid
  • [470] - Map Redice Monitor - Map Progress Grid
  • [469] - Map Redice Monitor - AngularJS Percentage Filter
  • [468] - Map Reduce Monitor - Monitored Jobs Grid
  • [467] - Add a general CSV parser operator to parse string, byte[] input to Map #451
  • [466] - Map Redice Monitor - AngularJS Util Service
  • [465] - Map Reduce Monitor - Unsubscribe Action
  • [464] - Map Reduce Monitor - Client-Side Settings
  • [463] - Map Reduce Monitor - WebSocket Unsubscribe
  • [461] - Map Reduce Monitor - Multiple Jobs Monitoring
  • [460] - Map Reduce Monitor - Progress Bar Animation
  • [459] - Map Reduce Monitor - Upgrade to AngularJS 1.2.4
  • [458] - added xml parser operator and its test, fixes #457
  • [456] - Github 444
  • [454] - Map Reduce Monitor - AngularUI Bootstrap
  • [453] - Map Reduce Monitor - Production Build with Grunt
  • [452] - Map Reduce Monitor - jshint
  • [451] - CSV input operator (CDR processing)
  • [450] - Map Reduce Monitor - Progress Bars
  • [449] - Map Reduce Monitor App - WebSocket Query
  • [448] - Map Reduce Monitor - Error Notifications with pnotify
  • [444] - Map Redice Monitor App - Publish Map/Reduce Updates as Array
  • [443] - Map Reduce Monitor App WebSocket Issue
  • [442] - Map Reduce Monitor - Node.js Proxy for Hadoop ResourceManager
  • [441] - Map Reduce Monitor - REST Service
  • [439] - Map Reduce Monitor - Server Configuration
  • [438] - Map Reduce Monitor - Settings
  • [436] - Map Reduce Monitor - Job Progress Grid
  • [435] - Map Reduce Monitor - WebSocket Service with AngularJS provider
  • [434] - Map Reduce Monitor - Unit Tests
  • [433] - Map Reduce Monitor - AngularJS Directives (widgets)
  • [432] - Map Reduce Monitor - Page Layout with Bootstrap
  • [431] - Map Reduce Monitor - Node.js Server
  • [430] - Map Reduce Monitor - Yeoman Generated App
  • [428] - Normalization operator (CDR processing)
  • [427] - Filter operator (CDR Processing)
  • [426] - Enrichment operator (CDR processing)
  • [425] - Aggregator operator (CDR processing)
  • [422] - Github 421
  • [421] - Create RedisOperator taking String,String for performance
  • [420] - Allow widgets to have adjustable height
  • [419] - DAG Styling, DAG Firefox Issue
  • [418] - Logical DAG - Firefox Bottom Margin Issue
  • [417] - Logical DAG Styling
  • [416] - Fix jquery build error
  • [415] - Organized scripts and server
  • [414] - fix #408, fix #413 Logical DAG - Show Stream Locality on Demand
  • [413] - Logical DAG - Right Aligned Legned and Show Locality Link
  • [412] - Improve Front Dev Environment
  • [411] - Improved dev environment for front
  • [410] - Map Reduce Monitor Web App
  • [409] - fix #393 Front Node.js Proxy
  • [408] - Logical DAG - Show Stream Locality on Demand
  • [407] - fixes #373
  • [401] - Physical DAG - Smart Zoom
  • [399] - Physical DAG - Bird's-Eye View
  • [393] - Front Node.js Proxy
  • [375] - fixes #367, improves reload time during dev on front
  • [374] - fixes #367, improves reload time during dev on front
  • [372] - Logical DAG Legend Styling
  • [371] - Logical DAG Legend
  • [370] - Fix issue316 issue317 pull
  • [369] - Logical DAG - Legend
  • [368] - Squashed commit of the following:
  • [367] - Precompile templates for better dev process
  • [366] - Documenting demos pull
  • [365] - Normalized all “processed” and “emitted” labels
  • [364] - Remove free memory from container metrics in UI
  • [362] - Dependency to dagre-d3 fork
  • [361] - Logical DAG - Stream Locality
  • [360] - Update physical operators collection to fetch from physical plan
  • [359] - Add source and sinks to physical operator list
  • [358] - Normalize processed/s emitted/s labels across data tables and dag view
  • [357] - Create Logical Operator Page
  • [356] - ContainerList view should show the log file name (stderr, stdout) in the info widget
  • [355] - fixes #349, recently-launched app does not request operator list
  • [354] - Make partitionable kafka input operator adjust partitions ifself for kafka partition change(del/add)
  • [353] - Upgrade kafka to 0.8 release
  • [352] - fixes #322
  • [351] - Non-partitioned operators
  • [350] - Key/Value lookup Storage Manager changes
  • [349] - Application launch error in console
  • [348] - Fixes #339, switches cluster metrics to websocket topic
  • [347] - fix #346 Physical DAG - Remove Container IDs
  • [346] - Physical DAG - Remove Container IDs
  • [345] - Added sensible default display for avg app age field in cluster metrics widget #341
  • [344] - Added build cmd to travis script, fixes #343
  • [343] - Build step for front not in travis script
  • [342] - Fix for #328
  • [341] - Cluster overview display items (initial launch)
  • [339] - Cluster stats should come from WebSocket topic
  • [337] - Add 1 to N partition support for kafka input operator with simple kafka consumer #311
  • [336] - logstream integration with siteops ui
  • [332] - Squashed commit of the following:
  • [329] - Documenting demos
  • [328] - Numbers in graph overlay illegible when close together
  • [325] - [DB Lookup & Caching] Create Mongo-db based DB lookup operator
  • [323] - fixes #322
  • [322] - Make connectionList property not null in Redisoutputoperator
  • [313] - Marking operators non-partitionable #312
  • [312] - Set partitionable to false for all operators which are cannot be partitioned
  • [311] - Dynamic partition kafka input operator to adapt to real time load
  • [298] - LogStream - siteops conversion - totals per second ( pages / bytes)
  • [297] - LogStream - siteops conversion - plot requests over time (per page / per server)
  • [296] - LogStream - siteops conversion - top 10 servers with 404 response (req/sec)
  • [295] - LogStream - siteops conversion - top 10 URL's with 404 response (req/sec)
  • [294] - LogStream - siteops conversion - top 10 clients downloading (bytes/sec)
  • [293] - LogStream - siteops conversion - top 10 client IP's (req/sec)
  • [292] - LogStream - siteops conversion - server load (req/sec/server)
  • [291] - LogStream - siteops conversion - top 10 URL's (req/sec)
  • [290] - Kafka POM WARN during build
  • [277] - Kafka input operator should destroy itself if the topic doesn't exist
  • [270] - [DB Lookup & Caching] Improvement to the design of existing Storage Manager
  • [269] - [logstream] integration with sitestats ui
  • [111] - SPOI-1191 - #comment Added StandardDeviationKeyVal operator and test class.
  • [104] - Map Reduce Monitor Application

Version 0.9.1

New Feature

  • [SPOI-377] - Input adapter for CouchDB
  • [SPOI-378] - Output Adapter for CouchDb
  • [SPOI-410] - Document and test download to work with Cloudera CDH5
  • [SPOI-945] - Add ability to specify application properties, DAG, operator and port attributes in stram configuration file.
  • [SPOI-1088] - Add mouseover for new graphing module
  • [SPOI-1172] - DAG Visualization - Partition Shape
  • [SPOI-1215] - Gauges in Container View
  • [SPOI-1407] - Add Jar File View
  • [SPOI-1415] - De-duplicaton operator
  • [SPOI-1483] - Hadoop management script or service
  • [SPOI-1501] - Widget - logical operators
  • [SPOI-1535] - Memory Gauge Widget
  • [SPOI-1536] - CPU Gauge Widget
  • [SPOI-1550] - Logical DAG - Multiple Metrics
  • [SPOI-1556] - Logical DAG Real-Time Metrics - Tracking Model Changes
  • [SPOI-1561] - Logical DAG - Throughput
  • [SPOI-1562] - Logical DAG - Prev/Next Metric Navigation
  • [SPOI-1600] - Logical DAG - Show Two Metrics
  • [SPOI-1235] - Add support for “DB lookup” functionality
  • [SPOI-1236] - Provide support for Cassandra NoSQL database

Bug

  • [SPOI-600] - Twitter demo failed when I did two simulteneouce operator/container kill
  • [SPOI-993] - Plan modification gives NPE when the containers aren't deployed yet
  • [SPOI-1280] - Recording Chart widget shows up with “X”
  • [SPOI-1303] - If the operator is setup sucessfully, ensure that it gets a chance to teardown.
  • [SPOI-1331] - When an operator is undeployed, its input streams should be disconnected from any upstream operators.
  • [SPOI-1388] - Nodes are showing IP instead of hostnames
  • [SPOI-1412] - Buffer Server uses a different version of guava base jar than Malhar
  • [SPOI-1427] - Accessing Demos From clean VM is not working
  • [SPOI-1430] - When uploading jars, meta information should be extracted and return error if jar is not valid
  • [SPOI-1433] - checkboxes in lists intermittently unresponsive
  • [SPOI-1452] - GetApp web service returns 500 for apps killed via UI
  • [SPOI-1457] - Kill container call returns 500 when request payload is not json
  • [SPOI-1458] - After finish uploading jar from the development tab, the jar list does not get updated until browser's “refresh” button is pressed
  • [SPOI-1459] - get recordings REST call returning null values
  • [SPOI-1497] - Change the names of nodes in hadoop to be just hostname with the domain name.
  • [SPOI-1498] - StreamList widget has “Name” for stream id
  • [SPOI-1499] - CPU usage on operatorList widget is routinely over 100%, sometimes 900%?
  • [SPOI-1548] - Logical DAG - Partition Count
  • [SPOI-1555] - Logical DAG: Partition-Count Metric
  • [SPOI-1563] - Moving average should be computed based on endWindow timestamps
  • [SPOI-1570] - Update sandbox to support Hadoop 2.2 and DataTorrent 0.9.x
  • [SPOI-1572] - gateway automatic update via websocket throws errors
  • [SPOI-1590] - Datanode does not successfully connect to cluster after initial startup
  • [SPOI-1614] - Front files not included during builds after migration to Malhar

Improvement

  • [SPOI-1005] - CLI: create a default clirc that is controlled by the installer
  • [SPOI-1099] - Display app master implementation version
  • [SPOI-1110] - BufferServer Stats should be passed as part of the port stats?
  • [SPOI-1305] - Support StringCodec for enums natively in Attributes
  • [SPOI-1306] - Reduce the verbosity of the Context interface by changing the requirement for defaultValue argument for attrValue
  • [SPOI-1421] - Improve default name for widgets
  • [SPOI-1453] - Publish recently ended apps on WebSocket topics along with running app(s)
  • [SPOI-1560] - “ERROR: mux is missing ...” messages emitted during “mvn test” on stram
  • [SPOI-1574] - Document the additions to the config file specification
  • [SPOI-1582] - Color non-empty filter fields for emphasis
  • [SPOI-1583] - Limit default number of rows for physical operators
  • [SPOI-1588] - Create service wrapper for datanode, namenode, nodemanager, resourcemanager, and historyserver
  • [SPOI-1592] - Latency Measure Units
  • [SPOI-1596] - Service wrapper for DTGateway
  • [SPOI-1601] - Create meaningful names to the demos instead of providing entire path as the demo names
  • [SPOI-1605] - Make default dashes into logical and physical for Instance
  • [SPOI-1606] - Remove various metrics from logical operator list

Task

  • [SPOI-907] - Create Application DAG visualization
  • [SPOI-1002] - List application names instead of class names where available
  • [SPOI-1140] - Annotate dag visualization with stream throughout and other data
  • [SPOI-1237] - Handle multi-container failure
  • [SPOI-1281] - Remove TopN Widget from dashboard
  • [SPOI-1282] - Update benchmarks
  • [SPOI-1297] - Recover from “multiple container” failure
  • [SPOI-1321] - Automate datanode provisioning
  • [SPOI-1354] - Figure out a way to have charts be dramatically different for key combinations
  • [SPOI-1360] - Create an automation framework for DataTorrent cloud provisioning
  • [SPOI-1365] - Add ability to create, delete, and bootstrap multiple nodes with a single command
  • [SPOI-1385] - Reduce memory used for Twitter demo containers
  • [SPOI-1411] - Deprecate old Malhar webdemos once logstream is available
  • [SPOI-1431] - GET /ws/v1/jars should return more meta information
  • [SPOI-1432] - Implement /ws/v1/jars/{jarname}/meta
  • [SPOI-1473] - Show main cluster stats in DT Console
  • [SPOI-1477] - Automate namenode provisioning
  • [SPOI-1478] - Recipe for Google Cloud provisioning
  • [SPOI-1479] - Base configuration and provisioning
  • [SPOI-1480] - Java install recipe
  • [SPOI-1481] - Recipe to install maven
  • [SPOI-1482] - Hadoop install recipe
  • [SPOI-1484] - Open source UI to Malhar
  • [SPOI-1518] - DT Console AngularJS Integration
  • [SPOI-1519] - Logstream - sitestats UI integration
  • [SPOI-1521] - Gateway to return error details upon status code 400s and 500s
  • [SPOI-1523] - Run certification tests on 0.9.0 release build
  • [SPOI-1524] - Automate performance benchmarking for different stream localities and event sizes
  • [SPOI-1526] - Update GC stram-site.xml to reflect new version settings and customizations
  • [SPOI-1527] - Automate stream locality perfromance benchmarking with the same tuple
  • [SPOI-1541] - Create clirc compatible with demos
  • [SPOI-1545] - Add documentation to chef repository
  • [SPOI-1553] - DataTorrent performance testing on Google Cloud
  • [SPOI-1554] - Create recipe for datatorrent local install
  • [SPOI-1589] - Create cluster creation helper script for gc-nodes
  • [SPOI-1591] - Create automatic installer for webapps
  • [SPOI-1496] - Website framework selection
  • [SPOI-1602] - Documentation index update to include Gateway API and Scalability Doc
  • [SPOI-1607] - Update docs for 0.9.1 release

Sub-task

  • [SPOI-1447] - Show stream throughput
  • [SPOI-1449] - DAG Visualization - Latency. Show latency per path/compute
  • [SPOI-1450] - Show partitionable operators
  • [SPOI-1464] - Allow operator to request/trigger repartitioning
  • [SPOI-1466] - DB lookup for MongoDB
  • [SPOI-1474] - Support operator specific stats in STRAM decisions
  • [SPOI-1485] - Add license headers to all relevant files
  • [SPOI-1486] - Move custom npm modules to node_modules folder, update npm registry
  • [SPOI-1487] - Update .travis.yml in malhar to also build and test Front
  • [SPOI-1488] - Create widget for displaying cluster info
  • [SPOI-1490] - Expose gateway API calls for getting cluster information
  • [SPOI-1529] - Understand exactly-once semantics.
  • [SPOI-1530] - Set-up cluster on local VM
  • [SPOI-1531] - Understand existing AdsDimension App
  • [SPOI-1532] - Develop similar app with at-least once semantics as AdsDimension App
  • [SPOI-1534] - Setup on Morado cluster for bench-marking
  • [SPOI-1593] - Move front folder into Malhar repository
  • [SPOI-1594] - Define code style standards, integrate a hinter
  • [SPOI-1595] - Create tests for more js files

GitHub - DataTorrent/Malhar

  • [335] - Fix issue316 issue317 pull
  • [331] - fix #330 Logical DAG Metrics
  • [330] - Logical DAG Metrics
  • [327] - Create a cluster stats widget
  • [326] - Refactored DB Lookup operators and Cache Manager
  • [324] - Add UI to repository
  • [321] - Merged malhar dev 0.9.1 to logstream
  • [320] - Squashed commit of the following:
  • [316] - Provide a property to set tuple size in the RandomWord input operator
  • [315] - fixes #314
  • [314] - RedisOutputOperator Exception
  • [310] - Fixing default redis listen ports to 6379. Fixes #309
  • [309] - Fix default redis listen port in webapps config.js
  • [308] - fixes #307
  • [307] - Change the format for passing multiple redis instances during partition
  • [305] - Fix github 278 and 283
  • [304] - Added DataCache, JDBCBasedCache Operator and its test case #303
  • [303] - [DB Lookup & Caching] Create an operator which caches database results in memory for specified time
  • [301] - Removing unused dependency and changing default daemon port to 9090.
  • [300] - Update default settings for webapps
  • [299] - A stream app to benchmark kafka partitionable input operator
  • [287] - Use Yammer Metric to tick the kafka consumer throughput
  • [284] - Squashed commit of the following:
  • [283] - Monitor throughput for each kafka consumer
  • [282] - Github 278 fix the Simple Kafka Consumer performance
  • [278] - A performance bug in Simple kafka consumer
  • [273] - Optimizing Redis Operator for inserting Maps
  • [264] - Added DataCache, JDBCBasedCache Operator and its test case
  • [262] - Marking operators non-partitionable
  • [260] - Figure out a way to have charts be dramatically different for key combinations in Machine Demo
  • [245] - Kafka Benchmark
  • [159] - Squashed commit of the following:
  • [147] - HighLow is not parameterized
  • [130] - CouchDB Output Adaptar
  • [289] - DimensionTimeBucket*Operator Usage and Issues
  • [252] - Web Apps - Karma Unit Tests
  • [251] - Machine Data Demo - Lookback Field Validation
  • [250] - Mobile Locator Demo - Phone Range Support
  • [232] - Show ingestion rate on MachineDemo UI
  • [104] - Map Reduce Monitor Application
  • [20] - Can not serialize Configuration Object

Version 0.9.0

Improvement

  • [SPOI-437] - Create more granular RESTful API calls in Stram for various entities in an instance
  • [SPOI-446] - Compute aggregate values for application instance on server side
  • [SPOI-465] - Increase unit test coverage to at least 50%
  • [SPOI-683] - Document Active MQ operatos in library/io/ActiveMQ*
  • [SPOI-1038] - Support more options in launch command
  • [SPOI-1127] - Standardize “subscribe” commands for all models and collections
  • [SPOI-1130] - Gateway to log requests and response (like access log in apache)
  • [SPOI-1150] - Create new mrcolor instance on init of topN visualization
  • [SPOI-1152] - CLI: add -v flag for debug logging
  • [SPOI-1154] - Provide physical plan connections via the Daemon API
  • [SPOI-1163] - Refactored pages and routing mechanism
  • [SPOI-1187] - Allow 50% width for adjacent widgets
  • [SPOI-1188] - Put widely used UI components in node_modules/datatorrent
  • [SPOI-1224] - CLI Help enhancement
  • [SPOI-1232] - Allow a way to assign a default value to an attribute while defining the attribute
  • [SPOI-1295] - Make UI compatible with Backbone v1.1.0
  • [SPOI-1333] - Show stream locality in Streams widget

New Feature

  • [SPOI-66] - App wide “do not autorecord” parameter
  • [SPOI-90] - Define and implement Apache server adapter node
  • [SPOI-117] - Boolean operator node
  • [SPOI-184] - Dashboard to have a page for one streaming application
  • [SPOI-389] - Parent jira for “alerts” as a product
  • [SPOI-444] - Provide a cascade merge unifier feature.
  • [SPOI-711] - Ads Dimension Time Bucket Missing Key
  • [SPOI-847] - Introduce Operator in Operator optimization (ThreadLocal)
  • [SPOI-916] - Create upload jar file widget
  • [SPOI-945] - Add ability to specify application properties, DAG, operator and port attributes in stram configuration file.
  • [SPOI-1007] - Run tests with headless browser for CI
  • [SPOI-1148] - Add CPU percentage to Operator List and Operator View
  • [SPOI-1213] - Map Reduce Debugger - AngularJS - WebSocket Integration
  • [SPOI-1227] - Map Reduce Debugger - JSHint
  • [SPOI-1248] - DAG Visualization - Physical View - Color Coding Containers
  • [SPOI-1323] - Pre-Deployment hook for Operator
  • [SPOI-1332] - Logical Plan Update According to new Daemon API
  • [SPOI-1345] - Add Alert Page - Application Name
  • [SPOI-1377] - Create central package for language items
  • [SPOI-1398] - Update DAG View Tool to Daemon API 0.9
  • [SPOI-1401] - Add AUTO_RECORD attribute on the operator level
  • [SPOI-1402] - Add capability to look up past alerts by timestamp
  • [SPOI-1235] - Add support for “DB lookup” functionality

Bug

  • [SPOI-440] - Change throughput calculation to use endWindow time
  • [SPOI-727] - Document demo for charts
  • [SPOI-904] - Tupleviewer: jump to valid index greater than total - visible fails
  • [SPOI-992] - Javascript operator scriptBinding not serializable
  • [SPOI-1037] - Daemon is creating a lot of threads that are not doing anything. Need to investigate and fix.
  • [SPOI-1107] - Send the Recording information as part of appropriate OperatorStats or PortStats
  • [SPOI-1164] - Mark non-partitionable operators on malhar
  • [SPOI-1186] - CSS issue when a page is not using dash manager
  • [SPOI-1189] - Misbehaving Mobile Locator Demo
  • [SPOI-1195] - DAG Explorer POC
  • [SPOI-1219] - Thread-Local streams should be marked as “inline”
  • [SPOI-1220] - Create a certification test for adsdimension demo
  • [SPOI-1221] - CLI should read configuration on app launch, not on start
  • [SPOI-1249] - duplicate sinks in streams in physical plan returned by web service
  • [SPOI-1283] - Fix mobile demo configuration in customer environment
  • [SPOI-1290] - Support hadoop.socks.server setting for dtcli
  • [SPOI-1298] - change product and package name and bump version
  • [SPOI-1299] - Documment current DT Phone Home data
  • [SPOI-1307] - Implement TupleRecorderCollection.deactivated method
  • [SPOI-1308] - Document how to configure applications
  • [SPOI-1318] - Logical DAG Display Issue in Firefox
  • [SPOI-1324] - MobileDemo: NullPointerException in StramChild
  • [SPOI-1327] - AtLeastOnceTest.testInlineOperatorsRecovery intermittent failure
  • [SPOI-1328] - Container not released and no operators running in it
  • [SPOI-1343] - Remove recordingNames from heartbeat
  • [SPOI-1349] - support set-pager command in dtcli
  • [SPOI-1351] - Need submit button and hour glass on machine data
  • [SPOI-1368] - Recording never loads in tupleviewer
  • [SPOI-1369] - Tests failing with new Daemon API changes
  • [SPOI-1371] - Add Alert Page fails
  • [SPOI-1374] - Drop twitter demo app from sandbox launch-demo jar
  • [SPOI-1386] - If I click on a “finished” task, the left top nav bar says “widget” instead of “applications”
  • [SPOI-1387] - If I click on a “finished” task, the left top nav bar says “widget” instead of “applications”
  • [SPOI-1391] - CPU percentage for operators are 100x too large

Task

  • [SPOI-634] - Enable unit tests in CI
  • [SPOI-716] - Compare original javascript with new script operators (Python, Bash)
  • [SPOI-720] - Coding conventions for Malhar GitHub
  • [SPOI-817] - Implement Log Input Operator
  • [SPOI-818] - Implement Parser Operator
  • [SPOI-824] - Document Daemon API
  • [SPOI-852] - Add life time per app on main dashboard
  • [SPOI-853] - Create Pie Chart widget based on D3.js
  • [SPOI-854] - Create Bar Chart widget based on D3.js
  • [SPOI-856] - Create Histogram (Real-Time Traffic) widget based on D3.js
  • [SPOI-857] - Site Stats Back-End Front-End Integration
  • [SPOI-861] - Redis Client with Node.js for Site Stats
  • [SPOI-862] - REST API with Node.js for Site Stats
  • [SPOI-907] - Create Application DAG visualization
  • [SPOI-954] - Publish Platform REST API
  • [SPOI-998] - Throughput computations should be controllable via attributes per application
  • [SPOI-999] - Throughput computations should be controllable per operator
  • [SPOI-1002] - List application names instead of class names where available
  • [SPOI-1033] - Implement OiO validations
  • [SPOI-1062] - Build out Google Cloud cluster
  • [SPOI-1066] - Design Versioning Scheme for forward and backward compatiblity
  • [SPOI-1086] - Dynamically scale up and down input Operators
  • [SPOI-1131] - Create Training Sessions
  • [SPOI-1133] - Create Training Session 2
  • [SPOI-1134] - Create Training session 3
  • [SPOI-1139] - Productize DAG visualization
  • [SPOI-1142] - Build a fraud detection poc
  • [SPOI-1143] - Implement new Daemon API
  • [SPOI-1159] - Conform to new Daemon API for REST and WebSocket
  • [SPOI-1160] - Move REST logic from DataSource to classes
  • [SPOI-1165] - Create a certification test for performance demo
  • [SPOI-1166] - Create a certification test for twitter demo
  • [SPOI-1168] - Create a certification test for mobile demo
  • [SPOI-1193] - replace web socket client in tuple recorder with async web socket client
  • [SPOI-1194] - remove “sync” from hdfs part file collection and implement web socket update of the newest part file
  • [SPOI-1197] - DAG View - Application Logical Plan as Standalone Java Application/Maven Plugin
  • [SPOI-1198] - DAG View - DAG Visualization from JSON file
  • [SPOI-1199] - DAG View - DAG Visualization from properties file
  • [SPOI-1207] - Common Apps Template
  • [SPOI-1210] - Map Reduce Debugger - AngularJS Directives
  • [SPOI-1216] - Benchmark performance benchmark with stream locality
  • [SPOI-1222] - CLI enhancements
  • [SPOI-1223] - CLI enhancements
  • [SPOI-1226] - Create list of platform attributes - port, operator, stream, application
  • [SPOI-1239] - Names for deamon and cli
  • [SPOI-1247] - Update UI for change in logicalPlan format
  • [SPOI-1256] - Make Machine Data Demo scalable
  • [SPOI-1262] - Parent JIRA for “face-lift demo apps for Hadoop World”
  • [SPOI-1265] - Web Demos - Common Application
  • [SPOI-1268] - Document all the attributes in Documentation (Guides)
  • [SPOI-1276] - Customer demo build tools
  • [SPOI-1277] - Add performance demo to customer apps
  • [SPOI-1279] - In Stream List view show DataLocality
  • [SPOI-1282] - Update benchmarks
  • [SPOI-1285] - keys in instanceinfo widget to be shortened
  • [SPOI-1286] - Add GB used to instanceOverview widget
  • [SPOI-1287] - Rename operator names and stream names on AdsCustomerApplication
  • [SPOI-1294] - Bump up the disk space in sandbox
  • [SPOI-1309] - Update Operations and Installation Guide by adding a section on parameter setting
  • [SPOI-1334] - Create data for Scalability White paper
  • [SPOI-1335] - Create Scalability White paper
  • [SPOI-1336] - Create First cut of logstream white paper
  • [SPOI-1337] - Create first cut for Gateway white paper
  • [SPOI-1339] - Productize LogStream white paper
  • [SPOI-1340] - Write Scalalability and Design Pattern white paper
  • [SPOI-1347] - Redis periodic cleanup node8
  • [SPOI-1348] - Add elapsed time to demo throughput widget
  • [SPOI-1352] - Need urls on Machine data demo to suppoer key combos
  • [SPOI-1353] - Machine data demo UI should show which keys are not selected
  • [SPOI-1356] - Make default look back on machine gen demo - 180 minutes
  • [SPOI-1362] - Set up separate Redis instance and application for Machine demo
  • [SPOI-1363] - gateway API recordings retrieval minor parameter name change
  • [SPOI-1366] - Reduce scope of dependency versions in Front
  • [SPOI-1367] - Name and ports not showing up in recording list
  • [SPOI-1372] - On dashboard change “Operations” to “Applications” in the first tab
  • [SPOI-1373] - Sandbox: size should be 8G by default, 4G causes most apps to not work
  • [SPOI-1375] - All sandbox apps must work in 8G VM. Need to test each
  • [SPOI-1381] - Add link to Malhar open source project in Sandbox
  • [SPOI-1382] - Add MachineCustomerApplication to launch-customer jar on node1
  • [SPOI-1399] - Add two new docs to the 0.9 release, to the website
  • [SPOI-1400] - Add timestamp for each tuple in tuple recordings

Sub-task

  • [SPOI-1109] - Change the frontend code to so that the recording information is received as part of the stats
  • [SPOI-1135] - Beautify “Twitter Top URLs” demo
  • [SPOI-1156] - mocha-phantomjs fails on test
  • [SPOI-1228] - Benchmark Performance with stream locality as thread-local
  • [SPOI-1229] - Benchmark Performance with stream locality as process local
  • [SPOI-1230] - Benchmark Performance with stream locality as node-local
  • [SPOI-1231] - Benchmark Performance with stream locality as rack-local
  • [SPOI-1240] - Create Hadoop 2.2 cluster on GC
  • [SPOI-1241] - Migrate morado cluster to Hadoop 2.1
  • [SPOI-1263] - Beautify “Mobile” demo app
  • [SPOI-1264] - Beautify “Machine Generated” demo app
  • [SPOI-1272] - Investigate Google Cluster Disks, Networking, Firewalls
  • [SPOI-1273] - Install Chef Server for central provisioning
  • [SPOI-1274] - Twitter demo - small changes in backend
  • [SPOI-1275] - Mobile demo - small changes in backend
  • [SPOI-1278] - Configure and install node0
  • [SPOI-1304] - Migrate core to Hadoop 2.2
  • [SPOI-1322] - Configure and install datanode
  • [SPOI-1379] - Configuration changes to support Hadoop 2.2
  • [SPOI-1380] - Fix customer demos creation for Hadoop 2.2
  • [SPOI-1394] - Enable testing during build for Malhar
  • [SPOI-1395] - Enable build testing for front
  • [SPOI-1396] - Enable build testing from Core

GitHub - DataTorrent/Malhar

  • [288] - Machine Demo Circular Keys. Squashed commit of the following:
  • [286] - Ads Dimensions Demo - REST API
  • [285] - Machine Data Demo - No Data Behavior
  • [281] - Machine Data Demo - Circular Keys Retrieval Unit Test
  • [280] - Machine Data Demo - Retrieving Circular Keys from Redis
  • [276] - Make topic and brokerList configurable properties for kafka input operator
  • [275] - fixed github issue #273
  • [274] - Couch operators 0.9
  • [267] - [logstream] add logstream usecases
  • [263] - Backend support to add a range of mobile numbers
  • [261] - Github issues #260 and #240
  • [259] - Make benchmarks appear more friendly in application list
  • [258] - Ads Dimensions Demo. Squashed commit of the following:
  • [257] - Machine and Ads Demos Common Styling
  • [256] - SPOI-1408 added AsyncHttpClient.close() calls
  • [255] - Github 240
  • [254] - Ads Dimensions Demo - Redis Unit Test
  • [253] - Port Ads Dimensions Demo to AngularJS
  • [249] - Added support to mobile app for adding range of numbers at a time #242
  • [248] - Remove reload button from MachineData demo
  • [247] - Web Apps Upgrade to Daemon API 0.9. Machine Demo optimizations. Squashed commit of the following:
  • [246] - Documentation had a typo which was repeated. I fixed it and also reorganized imports.
  • [243] - Web Apps - Upgrade to Daemon API 0.9
  • [241] - Add ads demo to common demo UI
  • [240] - Make the keys in Machine Data circular
  • [239] - Machine Demo - Negative Values in Random Data
  • [238] - Machine Demo - Load Indicator on Slow Response
  • [237] - Machine Demo - Instant Reload on Dimensions Change
  • [236] - Machine Demo - Line Chart Options for Empty Chart
  • [235] - Machine Demo - Server Polling Statistics
  • [231] - fixed github issue #220
  • [230] - Machine Demo - Error Handling
  • [228] - Machine Demo - Redis Query Optimization
  • [227] - Machine Demo - Client Cache for Chart Data
  • [226] - fixed github issue 219
  • [225] - 0.9 migration
  • [224] - Show hourglass on the machine data demo in case of network (internet) being slow
  • [223] - Setting Expiry Date for Redis Keys in RedisOutputOperator
  • [222] - Couch Input Output Adaptors
  • [221] - Web Apps - AngularJS Directive for Google Line Chart
  • [220] - Making Redis Operator partionable
  • [219] - Setting Expiry Date for Redis Keys in RedisOutputOperator
  • [218] - made output port of alert escalation autorecord
  • [217] - Add credit card fraud detection demo
  • [216] - Squashed commit of the following:
  • [215] - Machine demo should have 180 minutes as default in UI
  • [211] - Fraud App Rename
  • [210] - Web Apps - Fraud
  • [209] - 0.9 pull
  • [208] - Web Apps - Elapsed Time
  • [207] - 0.9 migration
  • [206] - SimpleMovingAverage is resetting the second last window state
  • [205] - Fix PythonOperator
  • [204] - Fix BashOperator
  • [203] - Marking operators non-partitionable
  • [202] - added new output port in JsonByteArrayOperator for emiting flattened map
  • [201] - #198 Machine Data Demo - Dynamic Dimensions. Squashed commit of the following:
  • [200] - added new output port in JsonByteArrayOperator for emiting flattened map
  • [199] - making the attributes configurable from stram-site
  • [198] - Machine Data Demo - Dynamic Dimensions
  • [197] - Web Apps Machine Demo Update/Multiple FIxes
  • [196] - Mobile Demo corrections
  • [195] - Web Apps Machine Demo Update/Multiple Fixes
  • [194] - Web Apps - Machine Data Demo - Last Minute Calculations
  • [193] - reduced the I/O worker thread multiplier to 1 as default in ning AHC
  • [192] - Web Apps - Express Version
  • [191] - Web Apps - Browser Dependencies
  • [190] - 0.9 migration
  • [188] - Web Apps (Demos) - Squashed commit of the following:
  • [186] - Web Apps - Styling
  • [185] - Web Apps - Readme
  • [184] - made the machine data and ads demo scalable
  • [183] - Web Apps - JSHint
  • [182] - Fixed seeds phone generator and other bugs with MobileDemo #172
  • [180] - suppress warning for stdout in console output operator
  • [178] - Web Apps - License Headers
  • [177] - Twitter demo application is called TwitterDevApplication
  • [176] - Removed phone.html that is not being used. Fixes #175.
  • [175] - Remove phone html that is not being used
  • [174] - Resolving mobile demo bug
  • [173] - bug in phonemovement
  • [169] - Code format changes to KryoSerializableStreamCodec
  • [168] - Web Apps - Demos Descriptions
  • [167] - javascript operator now serializable
  • [165] - Squashed commit of the following:
  • [164] - Operator and stream names of adsdimension application can be clearer
  • [163] - Web Apps - Mobile Locator Demo - Google Maps Marker Labels
  • [162] - Web Apps - Machine Data Demo - CPU/RAM/HDD Gauges
  • [161] - Web Apps - Machine Data Demo - Device ID Dimension
  • [158] - Made Mobile enhancements #157
  • [157] - Mobile Demo: Minor enhancements
  • [156] - Web Apps - Index Page
  • [155] - Web Apps - Machine Data Demo
  • [154] - Web Apps - Mobile Locator Demo
  • [151] - change name and jump version
  • [150] - Twitter multiplier variance #148
  • [149] - Checkpoint consumer offset #146
  • [148] - Twitter demo: change tweet multipler from 100 to Random(90-110)
  • [146] - Commit offset at checkpoint for kafka input operator
  • [145] - Web Apps POC
  • [144] - MachineData: bug in MachineInfoAveragingOperator
  • [143] - migrated to ning for web socket interface provider
  • [142] - 0.4 migration
  • [141] - upgraded framework version to 0.3.5
  • [140] - method rename in RedisOutputOperator #138
  • [139] - Add partitionable kafka input operator #113
  • [138] - Change selectDatabase in RedisOutputOperator to setDatabase
  • [137] - Couch DB Output adapter added #130
  • [136] - Reducing tuple blast size #135
  • [135] - MachineData: reduce the number of tuples generated by the random generator
  • [134] - Mapreduce Pull Request
  • [133] - [logstream] logstream app flow
  • [132] - Bug in UniqueKeyValCounter
  • [131] - Deleted old unique value count operator #129
  • [129] - Delete the old UniqueValueKeyVal operator and rename the new UniqueValueCount
  • [128] - Corrections to UniqueValueCount #127
  • [127] - UniqueValueCount improvements
  • [126] - Changed deprecated call to setInline to setLocality.
  • [125] - Changed setInline call to setLocality
  • [123] - Clean up unused libraries
  • [122] - squashed changed to UniqueCount #112
  • [121] - Operator that counts unique value per key #112
  • [120] - add all the locality modes to performance test
  • [119] - Squashed changes and fixed version #99
  • [118] - MachineData integrated operators #99
  • [117] - Adding Redis input operators #43
  • [116] - Logstream merge 43
  • [115] - Adding Redis input operators
  • [114] - Implement the automate partitioned kafka input operator(1:1) #113
  • [113] - Add partitionable kafka input operator (1:1)
  • [103] - move the test residue to maven target directory so that mvn clean can clean all the files
  • [102] - Sqaushed changes for KryoSerializableStreamCodec & Test #95
  • [101] - Pull request for #95
  • [100] - Pull request for issue #95
  • [99] - Add more statistics calculations to machine data
  • [98] - Issue 95
  • [97] - Add machinedata demo
  • [96] - Added output port to json byte array operator to emit JSONObjects too
  • [95] - Create an Abstract SteamCodec which can be used for custom partitioning and uses kryo serialization
  • [94] - added two output ports - outputMap, outputJSonObject
  • [93] - Upgrade Kafka to 0.8
  • [88] - starmcli not taking the jar passed during launching application
  • [73] - Upgrade our kafka operator api to be compatible with mvn released Kafka version 0.8
  • [70] - Clean kafka package in malhar library #53
  • [52] - addCombination doesn't work in DimensionTimeBucketOperator
  • [46] - SiteOps Dashboard Look and Feel
  • [45] - Move SiteOps Demo to Node.js
  • [44] - Sliding window aggregation computations operators
  • [43] - Create Redis Input Operator
  • [42] - Please pull my latest changes
  • [41] - Web Demos - License Headers
  • [40] - Move Ads Dimensions Demo to Node.js
  • [39] - AlertEscalationOperatorTest test failing
  • [38] - Move Mobile Demo to Node.js
  • [37] - Move Twitter Demo to Node.js
  • [36] - Move Twitter, Mobile and Ads Demo to Node.js - fixes #34
  • [35] - Create AMQP Input Operator
  • [34] - Move Twitter, Mobile and Ads Dimensions Demo to Node.js
  • [33] - Ads Dimensions Demo - Common Template
  • [32] - Common Assets for the Demos
  • [31] - Migrate deprecated setInline calls. fixes #16
  • [30] - bump master version to 0.3.5-SNAPSHOT
  • [29] - Javascript operator scriptBinding not serializable
  • [28] - Node.js Ads Dimensions Demo - “Play” mode
  • [27] - Node.js Ads Dimensions Demo - Node.js Daemon
  • [26] - Node.js Ads Dimensions Demo - Dynamic Port
  • [25] - Node.js Ads Dimensions Demo - Readme
  • [24] - Node.js Ads Dimensions Demo Rename
  • [23] - Mrapplication
  • [21] - keyvalpair doing hash on key and value
  • [18] - 404 error in the UI for logicalPlan
  • [17] - Emitted tuples by InputOperator incorrectly shown as zero
  • [16] - Adapt new stream locality API
  • [11] - Create an alerts demo for testing alerts
  • [4] - Make alert throttle operator window id based instead of wall clock time.

Version 0.3.5

Bug

  • [SPOI-349] - InlineStream undeploy error
  • [SPOI-766] - Process for certification of 0.3.3 launch on demo server
  • [SPOI-953] - Implement Exactly Once
  • [SPOI-959] - Operator Properties Widget
  • [SPOI-976] - Mobile apps keep dying
  • [SPOI-1013] - Cluster clocks are not in sync
  • [SPOI-1017] - Daemon stopped publishing over WebSocket
  • [SPOI-1022] - Alert is removed from the list on server failure
  • [SPOI-1023] - “Method Not Allowed” error when deleting an alert
  • [SPOI-1030] - DT Phone Home throw NullPointerException
  • [SPOI-1034] - 404 Not Found when getting alerts
  • [SPOI-1035] - Cannot create alerts with stramcli
  • [SPOI-1041] - Operator Properties fail to load
  • [SPOI-1051] - URLs templating for REST API
  • [SPOI-1075] - Container planned/alloc does not update in real time
  • [SPOI-1076] - Buffer server read is a flat line
  • [SPOI-1081] - Get common urls for all demos
  • [SPOI-1087] - Fix container chart
  • [SPOI-1089] - Widgets initializing twice
  • [SPOI-1090] - Add d3 to package.json deps
  • [SPOI-1091] - dashboard manager css issue with background color
  • [SPOI-1092] - Widget list not rendering on initial page load
  • [SPOI-1093] - Memory leak when switching between pages
  • [SPOI-1097] - dashboards not being saved
  • [SPOI-1101] - Explore Kibana 3 as UI option for site stats
  • [SPOI-1103] - Kill button doesn't work on applist palette
  • [SPOI-1104] - Mode switch separator in header still visible
  • [SPOI-1105] - shutdown application command requires empty data object
  • [SPOI-1114] - Test for AlertManagerTest.testAlertManager failed
  • [SPOI-1118] - Upgrade grid to 4GB containers
  • [SPOI-1121] - URL for deleting alert is incorrect
  • [SPOI-1123] - Add alert action not sending all parameters
  • [SPOI-1126] - Latency is whacked after adding alerts
  • [SPOI-1137] - DAG Page View - JavaScript Error
  • [SPOI-1144] - stram-site properties not applied to the operators (possibly?)
  • [SPOI-1145] - ConcurrentModificationException when used launch local in stramcli
  • [SPOI-1147] - Cannot kill “running” or relaunch “killed” application instance from AppLIst widget
  • [SPOI-1157] - NoSuchElementException in stramcli

Epic

  • [SPOI-870] - Alerts parent jira for phase I

Improvement

  • [SPOI-462] - Feature to specify that an operator cannot be partitioned.
  • [SPOI-523] - Document OS style guidelines
  • [SPOI-750] - Performance Metrics Widget Unit Test
  • [SPOI-937] - Incorporate jQuery UI into current dashboard
  • [SPOI-962] - Daemon REST API Conventions
  • [SPOI-963] - For exactly-once pick the correct checkpoint for recovery
  • [SPOI-979] - Need old versions of javadoc available online
  • [SPOI-983] - Front-End Release Branch
  • [SPOI-1019] - switch to jquery ui tooltip
  • [SPOI-1028] - Correctly order output port list for operator AlertThreeLevelTimedEscalationOperator
  • [SPOI-1078] - Add appInfo to apps.list WebSocket topic
  • [SPOI-1094] - Subscribe to app-specific websocket topic while on application
  • [SPOI-1102] - Alerts should be a list of js object literals
  • [SPOI-1106] - tables should resize as widget resizes by default
  • [SPOI-1119] - Remove Malhar subtree dependency from Core
  • [SPOI-1125] - Utilize new “optional” attribute with alert template parameters

New Feature

  • [SPOI-641] - Setting partition on operator not supporting partitioning must result in error.
  • [SPOI-810] - Create a line charting module using d3.js
  • [SPOI-831] - Create REST API call for uploading JAR files
  • [SPOI-832] - Create REST API call to retrieve all uploaded jar files
  • [SPOI-834] - Create “Top N” widget
  • [SPOI-874] - Provide description information for filter/throttle/action classes to be used by UI
  • [SPOI-940] - Alert List Widget - Actions (Add/View/Delete)
  • [SPOI-987] - Add stramcli tab completion for alias and macro
  • [SPOI-989] - Operator Properties Widget - Data Access Logic
  • [SPOI-1011] - AlertModel - Delete
  • [SPOI-1012] - Mock Node.js Server - Alert Delete
  • [SPOI-1014] - AlertCollection - Retrieve
  • [SPOI-1016] - AlertModel - Create
  • [SPOI-1018] - GitHub issue exporter for changelog
  • [SPOI-1024] - AlertModel - Unit Test
  • [SPOI-1025] - AlertCollection - Unit Test
  • [SPOI-1026] - OpPropertiesModel - Unit Test
  • [SPOI-1029] - Unit Testing - FakeXMLHttpRequest
  • [SPOI-1040] - Node.js back-end for Ads Dimensions Demo
  • [SPOI-1046] - Templatized Alert Creation Page
  • [SPOI-1055] - Node.js Redis Client (Ads Dimensions Demo)
  • [SPOI-1080] - Node.js Daemon
  • [SPOI-1085] - Unified Demos Page
  • [SPOI-1095] - Move Twitter Demo to Node.js
  • [SPOI-1096] - Move Mobile Demo to Node.js
  • [SPOI-1111] - Get common urls for all demos - POC
  • [SPOI-1112] - Shutdown command from UI
  • [SPOI-1116] - Move Machine Gen Demo to Node.js

Task

  • [SPOI-843] - Evaluate current market features
  • [SPOI-846] - Implement exactly once
  • [SPOI-855] - Dynamic re-partitioning happens on a snapshot, need to do a moving average by default
  • [SPOI-895] - Implement REST API filterClasses for Alerts
  • [SPOI-896] - Implement REST API escalationClasses for Alerts
  • [SPOI-932] - CLI: support alert operations
  • [SPOI-941] - Include latency in operator list in the dashboard.
  • [SPOI-955] - Demo showing recurring payment check
  • [SPOI-956] - Demo application for Machine (appliance) generated data monitoring resource usage (resources like CPU, RAM, etc)
  • [SPOI-972] - Adapt core versioning scheme
  • [SPOI-973] - Setup node.js as a supported technology
  • [SPOI-982] - Version Eclipse settings
  • [SPOI-986] - Version Nebeans settings
  • [SPOI-995] - Unit test for AlertsManager
  • [SPOI-996] - Unit test for AlertEscalation Operator
  • [SPOI-1001] - Create a convenience script to start and stop hadoop cluster
  • [SPOI-1008] - Cluster configuration versioning
  • [SPOI-1009] - Need a signup link to Google group on Malhar first page
  • [SPOI-1010] - Notificaitons on Malhar to include Google group
  • [SPOI-1015] - Evaluate anomaly algorithms
  • [SPOI-1021] - Set up user communication process
  • [SPOI-1031] - Hide operations/development mode switch in UI
  • [SPOI-1039] - Grid access
  • [SPOI-1052] - Update node1 with latest release (0.3.4)
  • [SPOI-1053] - Add users to grid
  • [SPOI-1056] - Add create alert template REST call
  • [SPOI-1057] - Change create alert REST call to use a template and parameters
  • [SPOI-1058] - Retrieve alert REST call should include template name and parameters
  • [SPOI-1060] - Get rid of stramRoot from the REST API (back end)
  • [SPOI-1061] - Get rid of stramRoot from the REST API call (front end)
  • [SPOI-1064] - Make machine generated data demo generic and launch on node1
  • [SPOI-1070] - Dashboard look and feel for demos
  • [SPOI-1071] - Dashboard look and feel for twitter firehose demo
  • [SPOI-1072] - Dashboard look and feel for mobile demo
  • [SPOI-1073] - Dashboard look and feel for ads dimensional demo
  • [SPOI-1074] - Dashboard look and feel for machine gen demo
  • [SPOI-1077] - Dashboard look and feel for logs demo
  • [SPOI-1079] - Evaluate Integration with third party tools
  • [SPOI-1100] - Update https://github.com/DataTorrent/Malhar
  • [SPOI-1117] - Allow exactly-once with downstream at-most-once only
  • [SPOI-1132] - Create Training session 1
  • [SPOI-1151] - AngularJS Integration

Sub-task

  • [SPOI-876] - Allow for saving alert configurations as templates for future use, back-end
  • [SPOI-877] - Allow for saving alert configurations as templates for future use, front-end
  • [SPOI-978] - Fix dependency among open source and platform
  • [SPOI-1042] - create alert template model
  • [SPOI-1047] - create new page for add alert and re-route current url
  • [SPOI-1048] - create parameter fill-in fields
  • [SPOI-1113] - Remove child module poc build from framework build
  • [SPOI-1176] - Remove malhar subtree from Core

GitHub - DataTorrent/Malhar

  • [90] - Fixed generics usage and renamed operator classes as per convention
  • [89] - Added operator to convert json byte stream to java hash map
  • [87] - Added operator to convert json byte stream to java hash map
  • [86] - Web Demos Update
  • [85] - Web Demos - Architecture Documentation
  • [84] - Adding TravisCI build status to README #83
  • [83] - Add TravisCI Build status to README
  • [82] - Add TravisCI Integration
  • [81] - [Logstream] Reads apache logs from RabbitMQ and prints basic aggregations to console
  • [80] - Webdemos - round time to minute
  • [79] - Adding apps project and logstream application skeleton. #62
  • [78] - SiteOps Demo - Totals Calculation
  • [77] - Operator changes for issue #76
  • [76] - Remove hardcoded values from RabbitMQ input operator
  • [75] - [Issue 52]: addCombination doesn't work in DimensionTimeBucketOperator
  • [74] - AMQP input operator for logs with sample aggregations #35
  • [71] - Move Machine Generated Data Demo to Node.js
  • [70] - Clean kafka package in malhar library #53
  • [69] - Pull request for issue #53
  • [68] - Web Demos Update
  • [67] - SiteOps Demo - Redis Service
  • [66] - Web Demos - Describe Configuration in Readme
  • [65] - Ads Dimensions Demo - Redis Configuration
  • [64] - Web Demos - Relative URLs for JS/CSS
  • [63] - Adding apps project and logstream application skeleton. #62
  • [62] - Create skeleton application for log stream processing
  • [60] - Move SiteOps Demo to Node.js - Unit Tests
  • [59] - Move SiteOps Demo to Node.js - License Headers
  • [58] - Move SiteOps Demo to Node.js - Charts
  • [56] - Clean input operators in malhar-library
  • [55] - Github 54
  • [54] - PubSubWebSocket operators tests should be self contained and not need other helpers.
  • [53] - Clean up Kafka input/output operator
  • [51] - Ads webdemo is showing a drop in the graphs at the end
  • [50] - Link to webdemos in main readme
  • [49] - Clean input operators in lib
  • [47] - Ads Dimensions Demo - JavaScript Loading Issue
  • [22] - Node.js Ads Dimensions Demo
  • [15] - Cleanup malhar-library input operator packages

Version 0.3.4

Bug

  • [SPOI-569] - Ads demo charting is not stable; Needs cleanup
  • [SPOI-570] - Mobile demo does not run forever
  • [SPOI-599] - Sometimes in the metrics charts widget some charts are flat
  • [SPOI-617] - Change error code from 500 to 400 for bad request
  • [SPOI-630] - Stop recording action does not work
  • [SPOI-660] - Recording shows a red light, but no recording is being done
  • [SPOI-674] - AppModel and AppInstanceView need clean-up
  • [SPOI-704] - Red dots show up in the dashboard, as if the operator is recording
  • [SPOI-714] - Ads UI load is alow
  • [SPOI-715] - Ads demo chart is 25 mins behind
  • [SPOI-724] - Fix LocalFileInputOperator logic and documentation
  • [SPOI-730] - Developer version for Mac OS X
  • [SPOI-731] - Cannot record anymore. Tuple recorder is broken
  • [SPOI-732] - Cannot stop recording
  • [SPOI-742] - Installation/Compilation fails with maven-eclipse plugin
  • [SPOI-744] - Facilitate the “accepted” app state for the app instance view
  • [SPOI-747] - StreamingContainerManager.getContainerAgents() returns inactive containers
  • [SPOI-748] - Build Kestrel as 3rd party library and depend on it instead of including the code with operator library
  • [SPOI-751] - browser CPU utilization excessive with larger number of updates
  • [SPOI-755] - Evaluate ads dimensional data demo
  • [SPOI-756] - Account for single-item, non-list responses from web services
  • [SPOI-757] - docs directory re-link for new version install
  • [SPOI-762] - Application Model not updating correctly, causing graph issues
  • [SPOI-764] - Malhar engine build is including StramAppMaster in engine jar in some cases
  • [SPOI-765] - Datasource constructing tuple GET URL incorrectly for tupleviewer
  • [SPOI-768] - Log file rotation support issue
  • [SPOI-804] - If a node is bad the app master should choose a different node for the containers.
  • [SPOI-806] - list-apps shows duplicate entries
  • [SPOI-807] - downgrade jvm on node3
  • [SPOI-809] - Add pig distinct semantic operator to library.
  • [SPOI-811] - shutdown app does not work when not connecting to an app
  • [SPOI-842] - Generated javadocs are missing several packages
  • [SPOI-845] - Installer test fails when executing from different directory
  • [SPOI-849] - Start recording from port list
  • [SPOI-863] - Display application-wide latency in Application View
  • [SPOI-883] - StramDelegationTokenManager should not be started if security is not enabled
  • [SPOI-888] - Account for delay in Stram initialization in recently launched apps
  • [SPOI-890] - Stram unit test is creating events data in stram/stram
  • [SPOI-892] - getAppInfo throws an exception during the beginning of the application
  • [SPOI-902] - Tupleviewer filter by port fails
  • [SPOI-903] - Tupleviewer preview of loaded tuples not rendering
  • [SPOI-905] - Review/fix license headers in open source
  • [SPOI-911] - Sync recording call fails with 500 error
  • [SPOI-915] - Log collection tool for hadoop cluster
  • [SPOI-918] - Partitioning stopped on MobileLocatorDemo instance 48
  • [SPOI-933] - Specify application names in stram configuration file for the customer demos on the cluster.
  • [SPOI-946] - Certify Sandbox for Ubuntu
  • [SPOI-951] - Auto completed file name does not work with launch command
  • [SPOI-952] - Remove operator returns error about input stream still connected.
  • [SPOI-958] - LocalFsInputOperator test is failing
  • [SPOI-965] - UI: Make sure empty lists and single-item lists in JSON returned from Daemon is handled properly
  • [SPOI-969] - UI: Stop recording fails with error saying some recording name was not found with operator undefined.
  • [SPOI-971] - Aggregation function for application stats fails intermittently
  • [SPOI-974] - PageLoaderView Unit Test
  • [SPOI-981] - Ensure “ports” are in the operators of the logicalPlan response
  • [SPOI-991] - Daemon: windowIds must be sent as strings
  • [SPOI-997] - Compute throughput as rolling average

Improvement

  • [SPOI-434] - Ability to subscribe to buffer server stream at next begin window
  • [SPOI-450] - CLI: Allow wildcard in launch file name
  • [SPOI-463] - The appMetrics widget should remember the choices on the graphs (on/off)
  • [SPOI-470] - Provide allocated/planned container count
  • [SPOI-614] - Better Error Handling/Error Pages
  • [SPOI-620] - Allow recording from operator/port page
  • [SPOI-624] - Normalize appid/appId opid/operatorId and similar discrepancies
  • [SPOI-677] - Setup both Core and Front-End to run on local environment
  • [SPOI-685] - Explore other charting libraries to replace custom chart module
  • [SPOI-687] - Improve CSS for tupleviewer
  • [SPOI-696] - Only update whole application collection on applist page
  • [SPOI-712] - Explicit Errors when daemon provides data in unexpected format
  • [SPOI-753] - Overview metrics for Port View should have comma separations
  • [SPOI-835] - CLI needs to be able to handle spaces and escape characters
  • [SPOI-844] - operatorClass REST call should also accept “chart” or “filter” for charting and alert filters
  • [SPOI-901] - Add palette for containers list
  • [SPOI-938] - Centralize reused templates for links

New Feature

  • [SPOI-70] - Parent JIra: Webservices for Streaming Application
  • [SPOI-73] - Job completed report
  • [SPOI-74] - Reporting: Job completion report should have the list of persisted files per node
  • [SPOI-75] - Parameter to specify if persisted file names should be included in the job completion report
  • [SPOI-127] - CLI: Throughput data (streams)
  • [SPOI-129] - CLI: Streaming Ap Master data
  • [SPOI-134] - Webservice: Stream data/throughput per stream in the application
  • [SPOI-138] - Webservices: Data on committed windows
  • [SPOI-139] - Webservice: Latency across the dag/application
  • [SPOI-140] - Event Logging
  • [SPOI-143] - Webservice: Bottleneck analysis
  • [SPOI-219] - Evaluate if we need a naming scheme to identify a physical node of a logical dnode
  • [SPOI-337] - Do an in-node stream by user choice
  • [SPOI-403] - Design license file format
  • [SPOI-417] - Licensing Specification
  • [SPOI-460] - Create Preconfigured Sandbox
  • [SPOI-471] - Stream View
  • [SPOI-551] - Create Video for Recording Tuples
  • [SPOI-628] - Web service for event logging
  • [SPOI-698] - Add streamquery operators that support expression
  • [SPOI-699] - Add pass through streamquery operators
  • [SPOI-710] - DatatTorrent Wiki
  • [SPOI-718] - Implement at most once for Operators with two or more connected input ports.
  • [SPOI-719] - Implement Dag Validations for At-Most-Once
  • [SPOI-738] - DT Phone Home Phase II
  • [SPOI-829] - Calculate overall latency for application
  • [SPOI-830] - Deduce critical path in DAG for application
  • [SPOI-833] - operator's latency should be shown in the UI
  • [SPOI-891] - Create a web service API for describing operator class
  • [SPOI-906] - Add palette for ports list
  • [SPOI-908] - Selecting recordings from recording list fails
  • [SPOI-909] - CSS issues with tupleviewer when port name too long
  • [SPOI-920] - Application DAG UI Integration
  • [SPOI-921] - Sandbox icons
  • [SPOI-934] - Alert List Widget - Date/Time
  • [SPOI-936] - Alert List Widget - Dynamic Width
  • [SPOI-949] - Documentation processor from markdown to html

Task

  • [SPOI-60] - Streaming app master logging
  • [SPOI-191] - Enable compression for end of window data or for throttled N tuples (N sent together)
  • [SPOI-198] - Evaluate if buffer server should retain data in a compressed state
  • [SPOI-199] - Evaluate if we need buffer server to keep data in compressed state
  • [SPOI-425] - Checkpointing for exactly once operator recovery
  • [SPOI-514] - Open Source Transition for Library/Contrib/Demos
  • [SPOI-603] - Evaluation version for Mac OS X
  • [SPOI-631] - Make a site operations video
  • [SPOI-633] - Need changelog for new version
  • [SPOI-656] - Performance metrics charts does not remember preferences
  • [SPOI-681] - Added library description to index.html
  • [SPOI-682] - Display Application DAG in UI (evaluation)
  • [SPOI-706] - Allow app name to be specified in launch time config file
  • [SPOI-708] - Set up demo procedures
  • [SPOI-713] - Technical evaluation of UIs
  • [SPOI-717] - Create launch macros on demo server
  • [SPOI-734] - Push 0.3.3 to demo server
  • [SPOI-735] - Launch the latest software on customer app server
  • [SPOI-736] - Certify customer application server
  • [SPOI-737] - Clean up redis history on customer apps
  • [SPOI-740] - Updated documentation generation process
  • [SPOI-745] - DT Phone Home Server side work
  • [SPOI-754] - Create other user for node0 launches
  • [SPOI-759] - Technical Evaluation
  • [SPOI-760] - Format comments for ASF project for Malhar GitHub
  • [SPOI-761] - Format comments in API
  • [SPOI-769] - Fix sql DeleteOperator to make it pass thru.
  • [SPOI-770] - Fix SelectOperator to make it pass thru.
  • [SPOI-771] - Fix sql UpdateOperator code to make it pass thru.
  • [SPOI-773] - Add seolect expression index.
  • [SPOI-774] - Fix sql InnnerJoin operator join condition.
  • [SPOI-775] - Fix sql Outer Join operator to merge left/right/full join sql semantic.
  • [SPOI-776] - Add sql Having semantic operator to library.
  • [SPOI-777] - Add sql select top operator semantic to library.
  • [SPOI-778] - Add sql select between condition sematic to library.
  • [SPOI-779] - Add sql select compound condition AND/OR semanticf to library.
  • [SPOI-780] - Add sql in condition semantic to library
  • [SPOI-781] - Add sql like condition semantic to library.
  • [SPOI-782] - Add having coprae value semantic to library.
  • [SPOI-783] - Add having condition interface to support sql having operator semantic.
  • [SPOI-784] - Add sql select unary expression semantic to library.
  • [SPOI-785] - Create sql binary expreesion index semantic in library.
  • [SPOI-786] - Add sql select negate index semantic to library.
  • [SPOI-787] - Add sql slect sum index semantic to library.
  • [SPOI-788] - Add sql select string mid index semantic to library.
  • [SPOI-789] - Add sql slect string len semantic to library.
  • [SPOI-790] - Add sql select string upper/lower semantic to library.
  • [SPOI-791] - Add sql round double semantic to library.
  • [SPOI-792] - Add sql select rtound double semantic to library.
  • [SPOI-794] - Add sql sleect count aggregate semantic to libarry.
  • [SPOI-795] - Add sql select min/max fucntion semantic to library.
  • [SPOI-796] - Add sql select first/last aggregate semantic to library.
  • [SPOI-797] - Add sql sleect sum aggregate semantic to library.
  • [SPOI-798] - Add pig group operator semanitc to library.
  • [SPOI-799] - Add PIG filter operator semantic to library.
  • [SPOI-800] - Add PIG cross operator semantic to library.
  • [SPOI-801] - Add PIG split operator semantic to library.
  • [SPOI-812] - Design wireframes for widget for SiteStats
  • [SPOI-814] - Prototype UI dashboard change to facilitate “app store” paradigm
  • [SPOI-815] - Add pig join(inner) semantic operator to library.
  • [SPOI-816] - Add pig join(outer => left/right/full) semantic operator to library.
  • [SPOI-820] - Implement Dimension Operator that allows custom explosion
  • [SPOI-825] - Evaluate non-Hadoop Streaming Platforms IDE
  • [SPOI-826] - Add pig order by operator semantic to library.
  • [SPOI-827] - Add pig limit operator semantic to library.
  • [SPOI-828] - Add pig stream operator semantic to library.
  • [SPOI-836] - Design Alert-related API
  • [SPOI-840] - Add spark add flat map function semantic to operator library.
  • [SPOI-848] - Introduce ContainerLocal as replacement for Inline
  • [SPOI-850] - Evaluate Hadoop IDE
  • [SPOI-851] - Evaluate JavaScript data visualization libraries
  • [SPOI-858] - Class loader support issue: org.fusesource.hawtbuf.UTF8Buffer.class
  • [SPOI-859] - Evaluate Streaming Platform Back-End Development Workflow
  • [SPOI-860] - Evaluate Node.js as Back-End for Site Stats Demo
  • [SPOI-864] - Redesign REST for altered escalation approach
  • [SPOI-865] - Create “List of Alerts” widget
  • [SPOI-866] - Add DataSource methods for alert REST API
  • [SPOI-879] - Create a data list of grid nodes
  • [SPOI-881] - Create Compute-Local api
  • [SPOI-882] - Add compute local api to documents
  • [SPOI-884] - Update dhcp configuration on cluster nodes.
  • [SPOI-885] - Create prototype with Node.js + Redis + REST for Site Stats
  • [SPOI-886] - Implement remove logical operator
  • [SPOI-889] - Alerts persistence with Backbone.js models
  • [SPOI-893] - Externalize UI settings
  • [SPOI-894] - Implement JavaScript Filter operator for Alerts
  • [SPOI-910] - Node.js Mock Server and JSONP Cross-Domain Requests
  • [SPOI-913] - Flesh out “kill app” command from the instance view
  • [SPOI-914] - Alerts REST API Error Handling
  • [SPOI-935] - Make patch for flawed start/stop recording mechanism
  • [SPOI-939] - Remove references to unfinished features for 0.3.4 release
  • [SPOI-943] - Certify Sandbox on various OS
  • [SPOI-944] - Certify Sandbox on Mac
  • [SPOI-947] - Certify Sandbox for Windows
  • [SPOI-948] - Improvements in sandbox based on OS certification feedback
  • [SPOI-957] - Update license header
  • [SPOI-970] - Download latest G! documents
  • [SPOI-985] - Macro argument expansion in stramcli
  • [SPOI-1006] - Update @since tags for 0.3.4

Sub-task

  • [SPOI-880] - Investigate options for sandbox environment
  • [SPOI-922] - Sandbox default hadoop configurations
  • [SPOI-923] - Sandbox demo script and application launcher
  • [SPOI-924] - Sandbox size and performance optimizations
  • [SPOI-926] - Sandbox documentation
  • [SPOI-942] - Demo launch page
  • [SPOI-990] - Add launch-demo macro to clirc during install
  • [SPOI-1004] - End user license agreement updates

GitHub - DataTorrent/Malhar

  • [12] - Site Stats Operator and TopNOperator issues
  • [10] - getTopN() function returns ArrayList in com.datatorrent.lib.util.TopNSort class
  • [9] - Another Bug in offer(E e) function of com.datatorrent.lib.util.TopNSort.java
  • [8] - Bug in offer(E e) function of com.datatorrent.lib.util.TopNSort.java
  • [7] - Add ability to configure timeout of RedisOutputOperator
  • [6] - Add a continue on error functionality to AbstractKeyValueStoreOutputOperator
  • [5] - Add rollback to AbstractKeyValueStoreOutputOperator.
  • [3] - Map Reduce job tracker
  • [2] - Fix application name of mobile demo
  • [1] - Console Operators

Version 0.3.3

Bug

  • [SPOI-418] - Duplicate - Demos need better documentation
  • [SPOI-438] - Remove outdated zookeeper dependency from contrib test scope
  • [SPOI-456] - CLI: kill-container should accept container number
  • [SPOI-464] - Update README to clarify development environment sydtem requirements
  • [SPOI-472] - dashboard fails with empty app list
  • [SPOI-478] - Recreate a page view on URL change
  • [SPOI-509] - Fix UI build on Linux
  • [SPOI-517] - Ensure inputPorts and outputPorts are not undefined before merging them
  • [SPOI-519] - Investigate Intermittent cease of publishing from Daemon
  • [SPOI-527] - divison by zero exception in StreamingContainerManager for calculating throughput
  • [SPOI-554] - Fix MergeSort operator and test
  • [SPOI-571] - Siteops demo hangs
  • [SPOI-604] - Empty site subdirectory in dist installer
  • [SPOI-607] - Auto publish websocket data get stuck
  • [SPOI-618] - Start and stop recordings broke from DataSource change
  • [SPOI-622] - Get Port REST request fails for running applications after 399
  • [SPOI-629] - Fix DataSource.stopOpRecording usage (options object).
  • [SPOI-653] - Fix name in package.json
  • [SPOI-655] - Fix “Cannot read property ‘ports’ of undefined” bug with RecordingModel
  • [SPOI-659] - PortPageView should override the cleanUp method to unsubscribe to port topic
  • [SPOI-669] - Daemon starts secondary process if already running
  • [SPOI-673] - Operator List Widget only needs appId, not whole app model instance
  • [SPOI-733] - Incorrect documentation in README.txt developer version

Improvement

  • [SPOI-454] - CLI: Feedback when not connected to app
  • [SPOI-467] - Backbone.js MVC: View -> Model -> Data Source
  • [SPOI-469] - Normalize naming convention for all modules
  • [SPOI-500] - Move require calls to widget classes to the top of each page file
  • [SPOI-510] - Change API for Datasource module
  • [SPOI-512] - Remove dataSource from model attributes
  • [SPOI-528] - PortInfoWidget Unit Test
  • [SPOI-547] - Convert Notifier module to an Object
  • [SPOI-563] - Move WebSocket creation out of DataSource constructor
  • [SPOI-582] - Document Front-End Architecture (UML Diagrams)
  • [SPOI-596] - Move require calls to the top of pages.js
  • [SPOI-602] - Get siteops demo to work without needed high bandwidth for UI
  • [SPOI-608] - Rename Page Views according to naming conventions
  • [SPOI-615] - Add ‘port’ to breadcrumb label in port view
  • [SPOI-616] - The ‘type’ attribute in portmodel is not being extrapolated in subscribeToUpdates
  • [SPOI-621] - The DataSource should URL-Encode port name in the getPort method
  • [SPOI-676] - Return better response for apps.list when no apps running

New Feature

  • [SPOI-135] - Webservice: Provide statistics per streaming operator
  • [SPOI-397] - Download Parent Jira
  • [SPOI-402] - Document demo examples
  • [SPOI-460] - Create Preconfigured Sandbox
  • [SPOI-466] - Design and Implement Port View
  • [SPOI-468] - Design and Implement Container View
  • [SPOI-476] - DT phone home first cut
  • [SPOI-503] - Create a pipeline for handling UI feedback
  • [SPOI-518] - Implement at most once for Operators with one or zero connected input ports.
  • [SPOI-598] - Daemon to serve historical stats data
  • [SPOI-700] - Remove demo/groupby

Sub-task

  • [SPOI-453] - Update the mobile demo to version with map
  • [SPOI-513] - Move JavaSerializationStreamCodec operator to library/util package
  • [SPOI-515] - GitHub Release: Review/format source and docs for library/io/AxctiveMQ Input Operator.
  • [SPOI-516] - GitHub repository structure and build system changes
  • [SPOI-520] - Test strategy for operators that currently use LocalMode
  • [SPOI-521] - Review/fix code formatting / style issues
  • [SPOI-524] - Refactor script operators
  • [SPOI-526] - Modify library to use CollectorTestSink
  • [SPOI-539] - Move PerformanceTestCategory annotation to library
  • [SPOI-541] - Port View Unit Tests
  • [SPOI-542] - Publish library tests jar
  • [SPOI-543] - Remove dependency to Tuple for library tests
  • [SPOI-545] - DataSource Unit Tests
  • [SPOI-546] - Determine best approach to mocking require()d modules
  • [SPOI-552] - RedisOutputOperator stops running abruptly
  • [SPOI-553] - Test code for InnerJoin/InnerJoin2 operators
  • [SPOI-555] - Fix Unifier in match/change operators
  • [SPOI-556] - Graphs show up after a delay with the new ads demo web changes.
  • [SPOI-558] - Benchmark tests depend on STRAM
  • [SPOI-559] - Move benchmark tests to a separate module
  • [SPOI-560] - GitHub release: SumTest math operator still has reference to STRAM
  • [SPOI-561] - Duplicate - Separate benchmark tests into a separate module
  • [SPOI-562] - Fix HdfsOutputTest to not depend on STRAM
  • [SPOI-565] - GitHub release: Change Http operator tests to use mortbay jetty
  • [SPOI-566] - Remove reference to StramTestSupport from KafkaInputOperatorTest
  • [SPOI-567] - KafkaInputOperatorTest is using DAG
  • [SPOI-568] - Modify contrib build to not depend on STRAM
  • [SPOI-583] - Implement a sample PubSubWebSocket servlet for testing
  • [SPOI-584] - Contrib test classes have dependency to bufferserver
  • [SPOI-585] - Implement a helper OperatorContext
  • [SPOI-586] - PageLoaderView Unit Tests
  • [SPOI-587] - Create SQL operator base interface/class
  • [SPOI-588] - Add SQL Select Oprator to library
  • [SPOI-589] - Create Sql Update operator in library
  • [SPOI-590] - Create Sql Delete operator in library
  • [SPOI-591] - Create SQL GourpBy/OrderBy Operator in library
  • [SPOI-593] - Create SQL Outer join operator in library
  • [SPOI-605] - Port View - Info and Overview Widgets
  • [SPOI-636] - Apache Open Source Release : Review cods and source code for lirary/algo operator.
  • [SPOI-637] - Create output Unifier on library/algo/BottomNOperator.
  • [SPOI-639] - Create output port unifier for library/algo/Distinct Operator.
  • [SPOI-640] - Create output port unifier for library/algo/FirstMatchMap Operator.
  • [SPOI-642] - Create output port unifier for library/algo/FirstMatchStringMap Operator.
  • [SPOI-644] - Create output port unifier for library/algo/FirstN Operator.
  • [SPOI-645] - HttpOutputOperatorTest is failing
  • [SPOI-646] - Create output port unifier for library/algo/InsertSort Operator.
  • [SPOI-647] - NavModel/Router Unit Tests
  • [SPOI-648] - Remove hash map output port from Insert sort operator.
  • [SPOI-649] - Create output port unifier for library/algo/InvertIndex<K, V> Operator.
  • [SPOI-650] - Mocha global leaks issue
  • [SPOI-651] - Create output port unifier for library/algo/InvertIndexArray<K, V> Operator.
  • [SPOI-652] - SlidingWindowTest is failing
  • [SPOI-654] - Fix output port unifer for LeastFrequentKey operator.
  • [SPOI-658] - Add Mocha Console Reporter
  • [SPOI-661] - PageLoaderView Router Navigation Unit Tests
  • [SPOI-662] - Refactor Bash Script oPerator.
  • [SPOI-663] - Refactor Python script operator.
  • [SPOI-664] - Fix output port unifier for MostFrequentKey Operator
  • [SPOI-665] - Fix output port unifier for MostFrequentKeyValMap Operator
  • [SPOI-666] - Fix output port unifier for libarry/algo/TopN Operator
  • [SPOI-667] - Change output port for library/algo/TopNUnique Operator, add output port unifier.
  • [SPOI-668] - Remove demo operator TupleOperator from library.
  • [SPOI-671] - Create unifier on output port for library/algo/UniqueKeyValOperator
  • [SPOI-672] - Remove demo operator : libarry/algo/WindowHolder
  • [SPOI-675] - Review source/docs for library/io operators
  • [SPOI-684] - Remove stram dependency from hdfs input operator test.
  • [SPOI-686] - Fix HTTP output operator and test, failing right now.
  • [SPOI-688] - Remove TestTupleCollector from io tests, it is not a test for any thing.
  • [SPOI-689] - Remove com.library.io.anootation empty test package.
  • [SPOI-690] - Move com.library.io.helper test to sample library.
  • [SPOI-691] - Review source code/docs for library/logs operator.

Technical task

  • [SPOI-557] - Review/change code for MergeSort Operator

Task

  • [SPOI-205] - Protocol for stram to change network/message bus parameters of the outstream of output adapter dnode
  • [SPOI-324] - Performance data to be written to HDFS
  • [SPOI-459] - Test installation on OS X
  • [SPOI-473] - GitHub Release : Code review for library/math operators
  • [SPOI-474] - Document mobile demo
  • [SPOI-475] - Document twitter demo
  • [SPOI-477] - Review/format doc for apche open source release of library operators.
  • [SPOI-499] - Review/format doc for apche open source release of library stream operators.
  • [SPOI-504] - GitHub Release : Review cods and source code for lirary/multi window operator.
  • [SPOI-505] - GitHub Release: Review/Format source/docs/test for AbstractSlidignWindowKeyVal Operator
  • [SPOI-506] - GitHub Release : Review/Format source/docs/test for library/MultiSlidingWindowKeyVal Operator
  • [SPOI-507] - GitHub Migration: Review/Format source/docs/test for library/SimpleMovingAverage Operator
  • [SPOI-508] - GitHub Release : Review/Format source/docs/test for library/MultiSlidingWindowRangeKeyVal Operator
  • [SPOI-511] - GitHub Release : Review/Format source/docs/test for library/logs operator
  • [SPOI-514] - Open Source Transition for Library/Contrib/Demos
  • [SPOI-530] - Operator tests dependent on stram.
  • [SPOI-532] - Remove stram dependency from com.datatorrent.lib.testbench.EventGeneratorTest
  • [SPOI-533] - Remove stram dependency from com.datatorrent.lib.testbench.RandomEventGeneratorTest
  • [SPOI-534] - Remove stram depnedncy from com.datatorrent.lib.testbench.SeedEventGeneratorTest
  • [SPOI-535] - GitHub release - Remove stram dependency from com.datatorrent.lib.math.MaxKeyValTest
  • [SPOI-536] - Remove stram dependency from com.datatorrent.lib.math.MinKeyValTest
  • [SPOI-537] - Remove stram dependency from com.datatorrent.lib.math.MaxMapTest
  • [SPOI-540] - Remove stram dependency from com.datatorrent.lib.io.ActiveMQInputOperatorTest
  • [SPOI-549] - Create a process for change.log for .3.3 release
  • [SPOI-550] - Need license file changes to add DT Phone Home
  • [SPOI-564] - Duplicate - Need to update license text to reflect “DT Phone Home”
  • [SPOI-572] - Create a library for sql operators
  • [SPOI-573] - Move GroupBy to Sql library
  • [SPOI-574] - Move innerjoin operator to Sql lib
  • [SPOI-575] - Move util/DerbySqlStreamOperator to sql libraray
  • [SPOI-576] - Move util/AbstractSqlStreamOperator to sql library
  • [SPOI-577] - Move OrderByKey operator to sql library
  • [SPOI-578] - move algo/OrderbyKeyDesc operator to sql lib
  • [SPOI-579] - move algo/orderbyvalue to sql library
  • [SPOI-580] - move algo/orderbyvaluedesc to sql lib
  • [SPOI-581] - move algo/innerjoincondition to sql lib
  • [SPOI-592] - Create SQL Inner Join Operator in library
  • [SPOI-594] - Archive release builds
  • [SPOI-601] - Include twitter demo on demo server
  • [SPOI-623] - Formalize a release procedure for future release
  • [SPOI-625] - Add footer to guides
  • [SPOI-626] - Get site ops demo UI to scale
  • [SPOI-627] - Site ops demo UI changes
  • [SPOI-635] - Ensure notice in html javadocs
  • [SPOI-670] - Create output port unifier in libarry/algo/UniqueCounter Operator
  • [SPOI-692] - change UI version to match streaming platform version
  • [SPOI-697] - Demo of JIRA commits feature
  • [SPOI-702] - Create a google groups for GitHub project Malhar
  • [SPOI-705] - Setup a server for customer apps

Version 0.3.2

Bug

  • [SPOI-29] - Pig: A inner join node
  • [SPOI-55] - A library module/adapter or node for creating keys from general text
  • [SPOI-325] - need better error reporting, not sure what host it is trying to connect to
  • [SPOI-412] - UI update rate should be 1 sec (default) and allow customization
  • [SPOI-441] - Investigate javadoc errors in build
  • [SPOI-452] - recompile embedded zmq with 1.6
  • [SPOI-482] - Fix the NPM issue when installing UI in the build process
  • [SPOI-490] - Clear chart when switching between operators
  • [SPOI-497] - Application dashboard starts flickering if I change the column widths too often
  • [SPOI-498] - Update rate on UI - Default and Customize

Improvement

  • [SPOI-326] - Byte code obfuscation for future releases: Allow Malhar platform with premium features to be shared with early customers
  • [SPOI-447] - Ability to filter app list in the CLI
  • [SPOI-451] - Convert daemon stop/start into single service script
  • [SPOI-484] - Add server timestamp to containers info

New Feature

  • [SPOI-69] - Persistence Node: A node to persist/spool every window into storage
  • [SPOI-89] - Define and implement a RSS input adapter node
  • [SPOI-112] - Versioning: Protocol between StramChild to app master
  • [SPOI-121] - Parent jira for supporting Pig programming language
  • [SPOI-153] - Heartbeat message
  • [SPOI-201] - Parent jira for Checkpointing support in streaming platform
  • [SPOI-254] - Design a general purpose read from stream and write to hbase node
  • [SPOI-259] - Create db adapters - Need one jira for each db
  • [SPOI-330] - Add a sql operator
  • [SPOI-332] - Do y! finance demo: calculate last price, volume, time, charts, and moving averages
  • [SPOI-333] - Do a pi demo
  • [SPOI-334] - Write an RSS read operator
  • [SPOI-335] - Add persistance/recording for operator, tuples should be stored in-order
  • [SPOI-336] - Add persistance/recording for an port of an operator, tuples should be stored in-order
  • [SPOI-343] - Design Malhar Deamon for the UI
  • [SPOI-344] - Design for Live as well as Historical data access via daemon/stram
  • [SPOI-345] - New optimal buffer server to get around memory issues with netty
  • [SPOI-347] - Develop cli commands as a foundational part of an operating system
  • [SPOI-348] - Allow commands to run as a script through CLI
  • [SPOI-356] - Input adapter for DRUID
  • [SPOI-357] - Output Adapter for DRUID
  • [SPOI-362] - Create Alert operator that does moving average and alerts if that drops by more than X%
  • [SPOI-365] - Output Adapter for Redis
  • [SPOI-374] - Size limits on log files (STRAM mainly)
  • [SPOI-380] - Add security to STRAM
  • [SPOI-383] - Implement do-As
  • [SPOI-386] - Design latency computations
  • [SPOI-387] - Design CPU, Memory, and Network IO usage
  • [SPOI-388] - Added resource usage to stats and access via webservice
  • [SPOI-391] - Design and implement at least once
  • [SPOI-394] - Add input adapter for Redis
  • [SPOI-395] - Create an output adapter for Redis
  • [SPOI-398] - Test download on Amazon
  • [SPOI-405] - Design, document, and test download to work with Apache Hadoop
  • [SPOI-416] - Local only Download licensing
  • [SPOI-426] - Create an attribute that forces the checkpoint to align with application window boundary
  • [SPOI-479] - Design and implement ad data charting
  • [SPOI-481] - UI design for application platform charting (live and historical dimensional data)
  • [SPOI-485] - Placeholder for list of possible UI work for Summit
  • [SPOI-486] - Enable security in the UI (need to check that the user has permission)
  • [SPOI-487] - Design and implement new SVG charts
  • [SPOI-489] - Alerts
  • [SPOI-491] - Create login error pages for UI on secure cluster
  • [SPOI-492] - Design UI for resources and latency data
  • [SPOI-493] - Design and implement per operator view
  • [SPOI-494] - Design and develop the “Operations” dashboard
  • [SPOI-495] - The recording tab should show list of recordings by operator or by operator:port
  • [SPOI-496] - Securitiy UI (sign-in, get token from deamon, etc.)

Story

  • [SPOI-371] - Review Storm features vs our open source plan
  • [SPOI-372] - Ensure that premium feature implementation is hard for outsiders to do on the open source tree

Task

  • [SPOI-14] - Dynamic run time optimization framework: Load balancing and load shedding
  • [SPOI-39] - Should be able to track window id through the DAG
  • [SPOI-40] - Output messages of each node should be persisted/buffered
  • [SPOI-63] - Output Adapter logging
  • [SPOI-64] - Levels in logging
  • [SPOI-231] - Demo tweeter feed analysis on Hadoop 2.0 with basic streaming setup
  • [SPOI-273] - Setup CI and run existing demos to ensure that code does not break any
  • [SPOI-419] - Certify our build on Amazon
  • [SPOI-420] - Convert mobile location demo to a google maps demo
  • [SPOI-421] - Ensure that checkpointing happens at Application window boundary
  • [SPOI-422] - Allow an operator developer to enable checkpointing within an application window
  • [SPOI-423] - Have a “ALLOW_CHECKPOINT_WITHIN_WINDOW” attribute (default is FALSE)
  • [SPOI-424] - Document new checkpointing logic in app dev guide and op dev guide, add attribute to operations guide
  • [SPOI-442] - ASF License header for downloadable sources
  • [SPOI-443] - Packaging for dev and cluster versions
  • [SPOI-480] - Add aggregate i/o buffer server bytes for application level
  • [SPOI-483] - Tie up loose ends for summit version
  • [SPOI-488] - Remove failure count column from “Live” table