Chukwa Change Log

Release 0.6 - 10/12/2014

  NEW FEATURES

    CHUKWA-722. Added SolrWriter to stream data to SolrCloud.  (Eric Yang)

    CHUKWA-719. Added Kerberos support for HBaseWriter.  (Sreepathi Prasanna via Eric Yang)

    CHUKWA-715. Added Oozie Adaptor for collecting Oozie metrics.  (Sreepathi Prasanna via Eric Yang)

    CHUKWA-712. Implemented generic REST Adaptor for fetch data from web service.  (Sreepathi Prasanna via Eric Yang)

    CHUKWA-674. Integrated Chukwa collector feature to Chukwa Agent.  (Eric Yang)

    CHUKWA-705. Updated Chukwa to support JDK7 and updated to Hadoop 1.2.1 and HBase 0.96.1.1.  (Eric Yang)

    CHUKWA-699. Updated timeline widget stylesheet.  (Eric Yang)

    CHUKWA-697. Added generic heatmap REST API and visualization.  (Eric Yang)

    CHUKWA-695. Added ability to export widgets from graph explorer. (Eric Yang)

    CHUKWA-581. Added support for custom reducer package name. (Ivy Tang via Eric Yang)

    CHUKWA-669. JMX Adaptor.  (shreyas subramanya via asrabkin)

    CHUKWA-671. Json processors for processing JMX data from Hadoop, HBase and Zookeeper.  (shreyas subramanya via asrabkin)

    CHUKWA-635. Collect swap usage. (Eric Yang)

    CHUKWA-664. Added network compression between agent and collector. (Sourygna Luangsay via Eric Yang)

  IMPROVEMENTS

    CHUKWA-713. Improved error handling and test case for System Metrics.  (Shreyas Subramanya via Eric Yang)

    CHUKWA-721. Updated Chukwa document to reflect changes in Chukwa 0.6.  (Eric Yang)

    CHUKWA-718. Updated Chukwa Agent REST API document and generation method.  (Eric Yang)

    CHUKWA-710. Set TCP socket reuse option for server sockets. (Shreyas Subramanya)

    CHUKWA-698. Update RAT plugin to 0.10 release and fixed missing license header. (Eric Yang)

    CHUKWA-696. Improve hicc stylesheets for form input.  (Eric Yang)

    CHUKWA-693. Updated Hadoop, ZooKeeper and HBase version for 0.6.0 release.  (Eric Yang)

    CHUKWA-686. Added FileTailingAdaptorPreserveLines adaptor. (Sourygna Luangsay via Eric Yang)

    CHUKWA-648. Make Chukwa Reduce Type to support hierarchy format. (Jie Huang via asrabkin)

    CHUKWA-650. Re-configure Demux ReduceNumber without re-starting the DemuxManager service. (Jie Huang via asrabkin)
    
    CHUKWA-649. Handle pid-file missing case while stoping chukwa processes. (Jie Huang via asrabkin)

    CHUKWA-645. Updated Hadoop and HBase dependency. (Eric Yang)

    CHUKWA-630. Added chukwa user setup to rpm package. (Noel Duffy via Eric Yang)

    CHUKWA-639. Updated to HBase 0.92.1. (Eric Yang)

    CHUKWA-647. Spread out intermediate data with the same ReduceType into different Reduce Tasks. (Jie Huang via asrabkin)

  BUGS

    CHUKWA-730. Updated Solr schema to work with Solr 4.7.2.  (Eric Yang)

    CHUKWA-729. Downgrade solr version for JDK 6 support.  (Eric Yang)

    CHUKWA-728. Update start/stop scripts to essential components.  (Eric Yang)

    CHUKWA-726. Reduce SocketAdaptor test case timer delay from 120 to 10 seconds.  (Eric Yang)

    CHUKWA-725. Remove Solr config file from license scanning.  (Eric Yang)

    CHUKWA-716. Fixed undefined System Properties for test cases and test case dependency.  (Eric Yang)

    CHUKWA-708. Update website reference to top level project.  (Eric Yang)

    CHUKWA-694. Improve demux configuration processing in HBaseWriter.  (Eric Yang)

    CHUKWA-692. Fixed race condition of agent startup in TestExecAdaptor.  (Eric Yang)

    CHUKWA-691. Improve pid file handling. (Eric Yang)

    CHUKWA-565. Added support HBaseWriter support for TsProcessor. (Bill Graham via Eric Yang)

    CHUKWA-677. Added pid file check before starting processes. (Sreepathi Prasanna via Eric Yang)

    CHUKWA-676. Exclude temporarily data from release audit tools. (Eric Yang)

    CHUKWA-672. Fix adaptor not found in checkpoint after shutdown. (shreyas subramanya via asrabkin)

    CHUKWA-675. Bad sleeping condition in FileTailer.java (Sourygna Luangsay via asrabkin)

    CHUKWA-668. Fix infinite loop in FileTailingAdaptor. (Sourygna Luangsay via Eric Yang)

    CHUKWA-653. Added pre-compile JSP code profile for build process. (Jie Huang via Eric Yang)

    CHUKWA-642. Added regular expression validation. (Eric Spishak via Eric Yang)

    CHUKWA-632. Added getter method for startOffset in AbstractProcessor. (Abhijit Dhar via Eric Yang)

    CHUKWA-637. Fixed default cluster selection in hicc. (Eric Yang)

    CHUKWA-654. Added additional check for getting row key. (Saisai Shao via Eric Yang)

    CHUKWA-652. Fixed JSON error in utilization.descriptor. (Saisai Shao via Eric Yang)

    CHUKWA-646. Fix file rotation correctly for 0 byte rotated files. (Ivy Tang via Eric Yang)

    CHUKWA-643. Updated Jersey dependency. (Prakhar Srivastava via Eric Yang)

    CHUKWA-641. Fix stack trace for dumpChunk command when invalid regular expression is specified. (Eric Spishak via Eric Yang)

    CHUKWA-636. Updated dependency to HBase 0.92.0. (Eric Yang)

    CHUKWA-638. Fixed timezone offset in test case. (Eric Yang)

    CHUKWA-300. Updated adaptor list in document. (Eric Yang)

Release 0.5

  NEW FEATURES

    CHUKWA-590. Migrate from ant to maven. (Eric Yang)

    CHUKWA-599. Added Metrics 2 support for Hadoop 0.20.203+. (Eric Yang)

    CHUKWA-575. Added cluster summary pig script. (Eric Yang)

    CHUKWA-571. Add digested system metrics to SystemMetrics demux parser. (Eric Yang)

    CHUKWA-563. Added configuration to skip HBase schema validation for HBaseWriter. (Eric Yang)

    CHUKWA-558. Update default Chukwa configuration files to use SocketAdaptor for Hadoop metrics streaming. (Eric Yang)

    CHUKWA-527  Configuration spellcheck. (Ari Rabkin)

    CHUKWA-520. Implemented graphing rest api and graph explorer for plotting HBase data to graph.  (Eric Yang)

    CHUKWA-530. Implemented Syslog Adaptor for mappig Syslog facility name to Chukwa data type.  (Eric Yang)

    CHUKWA-419. Replaced system metrics collection with Sigar. (Eric Yang)

    CHUKWA-444. Added HBaseWriter for storing time series data in HBase for faster random read/write. (Eric Yang)

    CHUKWA-473.  Make default processor configurable. (Bill Graham via asrabkin)

    CHUKWA-479.  Support HTTP trigger actions (Bill Graham via asrabkin)

    CHUKWA-469. Add JMSAdaptor. (Bill Graham via asrabkin)

    CHUKWA-477. Support post-demux trigger.  (Bill Graham via Eric Yang)

  IMPROVEMENTS

    CHUKWA-629. Updated Hadoop dependency to Hadoop 1.0.0 (Eric Yang)

    CHUKWA-627. Updated DISCLAIMER.txt and LICENSE.txt (Eric Yang)

    CHUKWA-626. Updated NOTICE.txt to include dependency licenses. (Eric Yang)

    CHUKWA-624. Added md5 signature signing to maven build for release tarball. (Eric Yang)

    CHUKWA-623. Added build profile src for building release source tarball. (Eric Yang)

    CHUKWA-622. Updated HDFS instruction for collector configuration. (Eric Yang)

    CHUKWA-620. Updated code analysis and license check in build structure. (Eric Yang)

    CHUKWA-618. Added HICC programming guide. (Eric Yang)

    CHUKWA-522. Added HICC User guide. (Eric Yang)

    CHUKWA-615. Changed HICC default view to monitor Hadoop. (Eric Yang)

    CHUKWA-612. Convert Chukwa document from forrest format to apt format. (Eric Yang)

    CHUKWA-608. Added Quick start guide. (Ahmed Fathalla via Eric Yang)

    CHUKWA-605. Update directory structure to be aligned with Hadoop. (Eric Yang)

    CHUKWA-601. Removed legacy mysql upgrade script. (Eric Yang)

    CHUKWA-596. Updated schema retrieve REST API to use end time for scanning columns. (Eric Yang)

    CHUKWA-586. Updated HBase dependency to HBase 0.90.3. (Eric Yang)

    CHUKWA-569. Added temperory fix for load meta data faster for graph_explorer.jsp. (Eric Yang)

    CHUKWA-573. Setup ChukwaMetrics to send to port 9097 and use ChukwaMetricsProcessor for parsing. (Eric Yang)

    CHUKWA-572. Changed disk and network stats to gauge instead of counter. (Eric Yang)

    CHUKWA-554. Remove jdbc.conf dependency from Chukwa. (Eric Yang)

    CHUKWA-562. Added RegExDirTailingAdaptor. (Deepak Deshpande via Eric Yang)

    CHUKWA-474. Replaced JSON library with JSON-Simple. (Eric Yang)

    CHUKWA-560. Improve HBaseWriter performance.  (Jerome Boulon via Eric Yang)

    CHUKWA-555.  Update descriptor files to use HBase REST API. (Eric Yang)

    CHUKWA-557.  Improve metrics differentation between recordName and contextName and clean up chukwa metrics.  (Eric Yang)

    CHUKWA-547. Adjustable Axes on HICC graphs. (Michael Chang via asrabkin)

    CHUKWA-548. Gray out options with no data. (Michael Chang via asrabkin)

    CHUKWA-546. SocketAdaptor. (Eric Yang via asrabkin)

    CHUKWA-493. Change release artifact naming. (Eric Yang via asrabkin)

    CHUKWA-525. Extensible DirTailingAdaptor. (Jaydeep Ayachit via asrabkin).

    CHUKWA-531. Bundle HBase 0.20.6 in library path, and comment out test case for HBaseWriter for now. (Eric Yang)

    CHUKWA-524. Use TODO-HBASE-HOME and TODO-HBASE-CONF-DIR to control hbase location. (Eric Yang)

    CHUKWA-521. Changed default HICC port to 4080. (Eric Yang)

    CHUKWA-506. Added retry logic for MetricDataLoader (William Bajzek via Eric Yang)

    CHUKWA-507. Imrpoved url validation for JMSAdaptor. (Bill Graham via Eric Yang)

    CHUKWA-501. Newer version of Ivy, pull in newer deps. (Eric Yang via asrabkin)

    CHUKWA-504.  Ability to delete file in FileAdaptor after successfully uploaded.  (Kirk True via asrabkin)

    CHUKWA-495.  Support for pig 0.7. (Bill Graham via asrabkin)

    CHUKWA-490. Enabling mdl.xml to be split into multiple files for ease of maintenance.  (Kirk True via Eric Yang)

    CHUKWA-480.  Allow CreateRecordFile to read job confs. (Bill Graham via asrabkin)

    CHUKWA-471. Expose JobConf to Demux Processors. (Jerome Boulon via asrabkin)

    CHUKWA-472. Added ability to configure timestamp format for the default Time Series Processor.  (Bill Graham via Eric Yang)

  BUG FIXES

    CHUKWA-628. Updated LICENSE.txt and NOTICE.txt to meet IPMC request. (Eric Yang)

    CHUKWA-621. Implemented findbugs and pmd code analysis check. (Eric Yang)

    CHUKWA-619. Disabled trace method on Chukwa servlets. (Julio Conca via Eric Yang)

    CHUKWA-617. Fixed test case casting and race conditions. (Eric Yang)

    CHUKWA-616. Updated release note and change log generation. (Eric Yang)

    CHUKWA-614. Updated parent iframe resize code. (Eric Yang)

    CHUKWA-613. Removed Jersey 1.1 from source code library path. (Eric Yang)

    CHUKWA-611. Updated Chukwa tarball to include chukwa-0.5.0-client.jar. (Eric Yang)

    CHUKWA-606. Updated HICC to interface with data in HBase. (Eric Yang)

    CHUKWA-588. Updated hbase.schema to match Hadoop 0.20.205.0 metrics and job summary. (Eric Yang)

    CHUKWA-609. Map / to /hicc for HICC web server. (Eric Yang)

    CHUKWA-593. Fixed infinite loop archiving at midnight. (Sourygna Luangsay via Eric Yang)

    CHUKWA-602. Set svn ignore properties to ignore eclipse and maven files. (Eric Yang)

    CHUKWA-597. Close stderr and stdout after parent process detached.  (Eric Yang)

    CHUKWA-38.  Download Hadoop jar files from ivy.  (Eric Yang)

    CHUKWA-595.  Fixed buffering time calculation for Agent to send data to Collector.  (Eric Yang)

    CHUKWA-594.  Fix row key matching in HICC web interface. (Preetam Patil via Eric Yang)

    CHUKWA-592.  Enable test case for HBaseWriter. (Eric Yang)

    CHUKWA-591. Check for empty result from HBase Scanner. (Eric Yang)
 
    CHUKWA-585. Enable HICC server to bind to all IP addresses. (Eric Yang)
 
    CHUKWA-582.  Change MoveToRepository to ignore anything starting with _. (Ari Rabkin)

    CHUKWA-587. Improved default cluster selection code. (Eric Yang)

    CHUKWA-579. Package initial_adaptors config file for tarball. (Eric Yang)

    CHUKWA-577. Convert HadoopMetricsProcessor to use Simple JSON library. (Eric Yang)

    CHUKWA-574. Added locking for recording metrics through log4j. (Eric Yang)

    CHUKWA-568.  Moved from json.jar library to JSON Simple library for SystemMetrics demux parser and add tags column family. (Eric Yang)

    CHUKWA-231. Update default view to show graph explorer and remove deprecated widgets. (Eric Yang)

    CHUKWA-7. Handle pid in /tmp in case CHUKWA_HOME is not defined. (Ari Rabkin via Eric Yang)

    CHUKWA-553. Retain file filter argument for DirTailingAdaptor. (Deepak Deshpande via Eric Yang)

    CHUKWA-559. Switched unit test HICC port to 4080. (Eric Yang)

    CHUKWA-543.  Parse only .view files for processing user view files.  (Eric Yang)

    CHUKWA-552.  Handle % signs in series names. (Michael Chang via asrabkin)

    CHUKWA-544. HICC graph should handle single series or multiple series more gracefully. (Eric Yang via asrabkin)

    CHUKWA-447. Correct time units in ExecAdaptor. (Eric Yang via asrabkin)

    CHUKWA-542.  Disable hbase jar download from maven, and include hicc rest api in HICC web.xml (Eric Yang)

    CHUKWA-516. Downgrade to HBase 0.20.6 until 0.90 arrives. (Eric Yang)

    CHUKWA-496. Fixed config file reference in design.html  (Bernd Fondermann via Eric Yang)

    CHUKWA-510.  JMSMessagePropertyTransformer inserts "null" string for null JMS message property values  (Bill Graham via asrabkin)

    CHUKWA-505. Don't include duplicate classes in chukwa-core.jar. (Bill Graham via asrabkin)

    CHUKWA-487.   Collector left in a bad state after temprorary NN outage. (Bill Graham via asrabkin)

    CHUKWA-499.  JMSAdaptor misses last argument in some cases when parsing. (Bill Graham via asrabkin)

    CHUKWA-498.  Agent logs incorrect number of chunks sent. (Bill Graham via asrabkin)

    CHHUKWA-494.  NPE when no time range is specified. (Bernd Fondermann via asrabkin)

    CHUKWA-492. Correction to database setup instruction. (Bernd Fondermann via Eric Yang)

    CHUKWA-489. Fixed malformed SQL in MetricDataLoader when values contain single quote character. (Kirk True via Eric Yang)

    CHUKWA-488. Filter user customized jar file path URL.  (Kirk True via Eric Yang)

    CHUKWA-483. Use SIGTERM to stop chukwa, not signal 1. (Bill Graham via asrabkin)

    CHUKWA-4. Collectors don't finish writing .dones on close. (Ahmed Fathalla via asrabkin)

    CHUKWA-485. Fix TSProcessor handling of some dates. (Bill Graham via asrabkin)

    CHUKWA-478. Fixed TestSocketTee intermittent failure.  (Chris Douglas via Eric Yang)

Release 0.4 

  NEW FEATURES

    CHUKWA-445. Realtime display at collector. (asrabkin)

    CHUKWA-454. DirTailingAdaptor can filter files. (Gerrit Jansen van Vuuren via asrabkin)

    CHUKWA-449. Utility to generate sequence file from log file. (Bill Graham via asrabkin)

    CHUKWA-440. Enable addon jar file for Demux from Distributed Cache. (Eric Yang)

    CHUKWA-448. Write-ahead buffering for arbitrary adaptors. (asrabkin)

    CHUKWA-441. Added real time Hadoop activity monitor. (Eric Yang)

    CHUKWA-433. File-per-post writer for benchmark purposes. (asrabkin)

    CHUKWA-431. UDP Adaptor. (asrabkin)

    CHUKWA-46. Ability to allow only local connections to agent control port. (asrabkin)

    CHUKWA-427. Automate FSM state calculation and load to database. (Eric Yang)

    CHUKWA-424. FSMBuilder driver script. (Jiaqi Tan via asrabkin)

    CHUKWA-395. Support for generalized buffering of adaptor data. (asrabkin)

    CHUKWA-405  Add a "stop all" command. (asrabkin)
 
   IMPROVEMENTS

    CHUKWA-468. Use SIGHUP instead of SIGTERM for shutdown.  (Eric Yang)

    CHUKWA-470. License headers. (asrabkin)

    CHUKWA-458. Documentation fixes for 0.4 (asrabkin)

    CHUKWA-446. Refactor start/stop scripts. (Eric Yang)

    CHUKWA-450. Ability to turn off sort in dumpchunks. (asrabkin)

    CHUKWA-359. Report detailed error on adaptor start failure. (asrabkin)

    CHUKWA-420. Clean up adaptor stop methods. (asrabkin)

    CHUKWA-437. Clean up widget category labels. (Eric Yang)

    CHUKWA-421. Use modification time to detect rotation. (asrabkin)

    CHUKWA-432. PipelineableWriter becomes an abstract class. (asrabkin)

    CHUKWA-429. Update HDFS heatmap color with rainbow colors. (Eric Yang)

    CHUKWA-418. Standardize logging. Mark adaptor IDs. (asrabkin)

    CHUKWA-207. Store HICC dashboard data in HDFS. (Eric Yang)

    CHUKWA-415. Read initial adaptors at start every time. (asrabkin)

    CHUKWA-413. Improve admin guide. (asrabkin)

    CHUKWA-345. Remove redundant 'application' field from Chunk API. (asrabkin)

    CHUKWA-409. Make SocketTeeWriter work in single-stage pipeline. (Thushara Wijeratna via asrabkin)

    CHUKWA-408. Add optional metadata to real-time feed. (asrabkin)

  BUG FIXES

    CHUKWA-523. Fix reload collectors. (Eric Yang via asrabkin)

    CHUKWA-508. Include contrib directory build files.  (Bill Graham via Eric Yang)

    CHUKWA-465. Setup default value for Post Demux Data Loader.  (Ahmed Fathalla via Eric Yang)

    CHUKWA-467. Disabled war file extraction to /tmp. (Eric Yang)

    CHUKWA-456. Remove /tmp/chukwa/hicc during test phase to prevent bad data building up and prevent test case to run properly. (Eric Yang)

    CHUKWA-460.  Removed dependency of writer.hdfs.filesystem for demux phase. (Eric Yang)

    CHUKWA-466. Fix failing tests. (asrabkin)

    CHUKWA-457. Fix TestUserResource. (Eric Yang via asrabkin)

    CHUKWA-463. Fix pattern match for tags. (Guillermo Perez via asrabkin)
    
    CHUKWA-464. Fix spelling of processClusterDirectory. (Guillermo Perez via asrabkin)

    CHUKWA-456. Fixed failing unit test. (Eric Yang)

    CHUKWA-455. Updated slaves.sh to detect collectors list. (Guillermo Pérez)

    CHUKWA-453. Specify bash instead of sh in script headers. (Gerrit Jansen van Vuuren via asrabkin)

    CHUKWA-451. Shouldn't use ssh to start processes locally. (Bill Graham via asrabkin)

    CHUKWA-452. Fix support for using pig on final archives. (Gerrit Jansen van Vuuren via asrabkin)

    CHUKWA-443. Updated HDFS metrics logging filename. (Eric Yang)

    CHUKWA-442. Updated Apache license for HICC Rest API. (Eric Yang)

    CHUKWA-438. Fixed column resizing issue in HICC (Eric Yang)

    CHUKWA-439. Added blank image for HICC. (Eric Yang)

    CHUKWA-436. Include time widget related javascript in the dashboard. (Eric Yang)

    CHUKWA-430. Narrow down the list of demux output for FSM to improve processing time. (Eric Yang)

    CHUKWA-428. Revised location for Chukwa HDFS repository. (Eric Yang)

    CHUKWA-241. Revise chukwa-config.sh so that chukwa runs after building from source. (Eric Yang via asrabkin)

    CHUKWA-2. Removed the default /var/log/messages streaming. (Eric Yang)

    CHUKWA-414. Better error messaging for pid file errors. (Bill Graham via asrabkin)

    CHUKWA-411. Hosts for DataNode ClientTrace messages parsed incorrectly. (Jiaqi Tan via asrabkin)

    CHUKWA-229. Fix file descriptor leak in ExecAdaptor. (asrabkin)  

Release 0.3

  NEW FEATURES

    CHUKWA-344. State-Machine Generation for input to SALSA visualizations. (Jiaqi Tan via asrabkin)

    CHUKWA-384. Added REST API charting capability. (Eric Yang)

    CHUKWA-369. Tolerance of collector failures. (asrabkin)

    CHUKWA-368. New data integrity validation tool. (asrabkin)

    CHUKWA-383. Added embed mode for HICC.  (Eric Yang)

    CHUKWA-382. Added export button to export HICC graph as static image.  (Eric Yang)

    CHUKWA-366. Custom tags. (asrabkin)

    CHUKWA-358. Real-time monitoring at collector. (asrabkin)

    CHUKWA-352. Xtrace in contrib. (asrabkin)

    CHUKWA-346. Simplified sink archiver. (asrabkin)

    CHUKWA-343. Static HDFS Heatmap visualization.   (Jiaqi Tan via asrabkin)

    CHUKWA-342. Static swimlanes visualization widget.   (Jiaqi Tan via asrabkin)

    CHUKWA-299. Added HDFS Spatial heatmaps visualization. (Jiaqi Tan via Eric Yang)

    CHUKWA-185. Ability to tail a whole directory. (asrabkin)

    CHUKWA-280. Added end to end test to detect iostat overflow. (Eric Yang)

    CHUKWA-194.  Backfilling tools.  (Jerome Boulon via asrabkin)

    CHUKWA-253. Added aggregations by user. (Cheng Zhang via Eric Yang)

    CHUKWA-95. Added Web Service API to export data from database. (Terence Kwan via Eric Yang)

    CHUKWA-281. Created pig scripts to perform down sampling. (Cheng Zhang via Eric Yang)

    CHUKWA-20. Added pig support for ChukwaRecords. (Jerome Boulon via Eric Yang)

    CHUKWA-279. Added swim lane visualization for Hadoop job progress. (Jiaqi Tan via Eric Yang)

    CHUKWA-267. Added TaskLogAppender for collecting Task Tracker Log file. (Eric Yang)

    CHUKWA-71. Status/version command for chukwa agent. (asrabkin)

    CHUKWA-271. Added SQL client to HICC for SQL data exploration. (Eric Yang)

    CHUKWA-210. Add new job conf metrics as database columns. (Cheng Zhang via Eric Yang)

    CHUKWA-91.  Additional test case for acks. (asrabkin)

  IMPROVEMENTS

    CHUKWA-390. Improvements to asynchronous acknowledgement mechanism. (asrabkin)

    CHUKWA-397. Allow "all" as search pattern. (asrabkin)

    CHUKWA-393. Support using pig on Chunks. (asrabkin)

    CHUKWA-392.  FIFO queueing of threads in collector. (asrabkin)

    CHUKWA-388.  Clean up user interface color.  (Eric Yang)

    CHUKWA-387.  Summarize mode for dumpChunks should count bytes. (asrabkin)

    CHUKWA-379.  Refactor sender code. (asrabkin)

    CHUKWA-374.  Adaptor.getStatus() shouldn't throw exceptions. (asrabkin)

    CHUKWA-373.  Test code for backpressure. (asrabkin)

    CHUKWA-370.  Exec adaptor should commit immediately. (asrabkin)

    CHUKWA-367.  Print metadata in DumpChunks. (asrabkin)

    CHUKWA-74.  Agent shouldn't log exceptions for down collectors. (Rushin Barot via asrabkin)

    CHUKWA-365.  Improved DumpArchive tool. (asrabkin)

    CHUKWA-364.  Design and Architecture document in documentation. (asrabkin)

    CHUKWA-333.  Copy release notes from 0.2 forward to Trunk. (asrabkin)

    CHUKWA-362.  Archiver group-by-cluster conf name shouldn't be hardcoded. (asrabkin)

    CHUKWA-355.  Remove obsolete datacollection.protocol package. (asrabkin)

    CHUKWA-270. ChukwaAgentController no longer uses stdout. (Rushin Barot via asrabkin)

    CHUKWA-356. More powerful file extractor. (asrabkin)

    CHUKWA-350. Improve docs, add programmer guide. (asrabkin)

    CHUKWA-282. Demux should output detailed per-operation ClientTrace records (Jiaqi Tan via asrabkin)

    CHUKWA-341. Heap space in HICC Jetty is configurable. (Jiaqi Tan via asrabkin)

    CHUKWA-284.  Checkpoint includes last byte Acked, not sent. (asrabkin)

    CHUKWA-25.  Archiver can group by cluster name. (asrabkin)
    
    CHUKWA-308. Added capability to start hicc without tomcat. (Eric Yang)

    CHUKWA-326. If the chukwa records produce errors in post process, move to InErrorDirectory. (Jerome Boulon via Eric Yang)

    CHUKWA-318. Added check for disk capacity for collector to write. (Jerome Boulon via Eric Yang)

    CHUKWA-276. Fix hourly and daily rolling to use a single reducer. (Jerome Boulon via Eric Yang)

    CHUKWA-313. Removed the 10 second ack from SeqFileWriter. (Eric Yang)

    CHUKWA-5. Adaptors have durable names. (asrabkin)

    CHUKWA-280. Added end to end test to detect iostat overflow. (Eric Yang)

    CHUKWA-278. Improve post process manager and metric data loader to support data loading from pig aggregation. (Eric Yang)

    CHUKWA-263. Added ability to configure demux parameters during build time. (Jerome Boulon via Eric Yang)

    CHUKWA-195. Update demux compilation to use Hadoop 0.20 jar file. (Jerome Boulon via Eric Yang)

    CHUKWA-268. Expose adaptor manager interface, refactor Agent class hierarchy. (asrabkin)

    CHUKWA-259. More flexible CollectorStub. (asrabkin)

    CHUKWA-274. Improve rate calculation for accumulated values for visualization. (Eric Yang)

    CHUKWA-275. Add standard deviation to statics calculation for HICC charts. (Eric Yang)

    CHUKWA-137. Add collector IP to the dataSink fileName. (Jerome Boulon via asrabkin)

    CHUKWA-258. More detailed logging of SQL errors. (asrabkin)

    CHUKWA-30.  Remove HDFS flush and connection holding in Collector. (Jerome Boulon via asrabkin)

    CHUKWA-142. Remove add(Chunk) from SeqFileWriter.  (Jerome Boulon via asrabkin)

    CHUKWA-82.  Added min,max,average to HICC Charting widget. (Terence Kwan via Eric Yang)

    CHUKWA-219. Improved usability of host selector, and updated caching for host list. (Eric Yang)

  BUG FIXES

    CHUKWA-407. PatternSyntaxExceptions should be sent to client. (thushara wijeratna via asrabkin)

    CHUKWA-402. Added readme and license notice file. (asrabkin)

    CHUKWA-401. Fix race condition in Dir Tailing adaptor test case. (asrabkin)

    CHUKWA-398. Make ConstRateAdaptor more deterministic. (asrabkin)

    CHUKWA-386. Fix a bug in echo id to the browser for the iframe servlet. (Eric Yang)

    CHUKWA-380. FTA shouldn't emit empty chunks. (asrabkin)

    CHUKWA-378. Disable TestArchive unit test. (asrabkin)

    CHUKWA-377. Revised xtrace adaptor code. (asrabkin)

    CHUKWA-372. select_multiple_callback does not work in hicc (Eric Yang via asrabkin)

    CHUKWA-354. Fix corner case in SinkArchiver. (asrabkin)

    CHUKWA-334. Fix LocalHdfsMover exception handling. (Jerome Boulon via asrabkin)

    CHUKWA-335. Discard meta line in job history file. (Cheng Zhang via asrabkin)

    CHUKWA-330. Make separation between fixed adaptor parameters and optional adaptor parameters for generating adaptor id. (Ari Rabkin via Eric Yang)

    CHUKWA-37. Remove stale and unused bin scripts. (asrabkin)

    CHUKWA-331. Fixed regular expression for down sampling base on record type. (Cheng Zhang via Eric Yang)

    CHUKWA-329. Correction to user job summary script. (Cheng Zhang via Eric Yang)

    CHUKWA-328. Handle empty sequence file more gracefully for MetricDataLoader. (Cheng Zhang via Eric Yang)

    CHUKWA-327. Remove START_TIME="0", FINISH_TIME="0" from JobHistory. (Cheng Zhang via Eric Yang)

    CHUKWA-325. Removed hourly rolling flag from daily rolling. (Jerome Boulon via Eric Yang)

    CHUKWA-324. For 0 matched field, changed MDL to skip empty SQL statement. (Eric Yang)

    CHUKWA-321. Added 2 seconds sleep to avoid chukwa agent race condition. (Eric Yang)

    CHUKWA-319. Added cluster name to pig user aggregation script. (Cheng Zhang via Eric Yang)

    CHUKWA-316. Added explicit rule for MetricDataLoader to prevent loading of data. (Eric Yang)

    CHUKWA-42. Clean up logging of adaptor creation. (Ari Rabkin)

    CHUKWA-312. Fixed chukwa-pig.jar class path for pig aggregation. (Cheng Zhang via Eric Yang)

    CHUKWA-304. Propagate exception upward for database test cases. (Jerome Boulon via Eric Yang)

    CHUKWA-314. Added filtering for start_time to remove invalid value. (Eric Yang)

    CHUKWA-304. Propagate exception upward for database test cases. (Eric Yang)

    CHUKWA-309. Fix MDL configuration for mr_job_conf table. (Cheng Zhang via Eric Yang)

    CHUKWA-301. Set TestFileExpirationPolicy controller port to 0 to avoid race condition. (Jerome Boulon via asrabkin)

    CHUKWA-302. Removed hard coded dependency of Hadoop jar file. (Eric Yang)

    CHUKWA-298. Update branch 0.2 and trunk version number. (Eric Yang)

    CHUKWA-296. Fix jsp syntax error. (Terence Kwan via Eric Yang)

    CHUKWA-294. Added post process data loader configuration.  (Eric Yang)

    CHUKWA-10. Fix TestAgentConfig (asrabkin)

    CHUKWA-291. Append slash to collector paths. (asrabkin)

    CHUKWA-287. Fix collector crash on startup when no args. (Eric Yang via asrabkin)

    CHUKWA-273. Agents correctly use configured collector port numbers. (asrabkin)

    CHUKWA-216. postProcess.sh should use HADOOP_CONF_DIR to find hadoop-site.xml. (Jerome Boulon via asrabkin)

    CHUKWA-261. Added X axis label for non Time Series graphs. (Eric Yang)

Release 0.2 - Unreleased

Release 0.1.2 - 2009-05-14

  INCOMPATIBLE CHANGES

  NEW FEATURES

    CHUKWA-97.  Refactored FileTailers, added LWFTAdaptor. (asrabkin)

    CHUKWA-236. Added migration script for moving database schema for Chukwa 0.1.1 to Chukwa 0.1.2. (Eric Yang)

    CHUKWA-78.  Added down sample SQL aggregation for job data, task data and utilization data. (Eric Yang)

    CHUKWA-17. Collect PS output every 10 minutes and parse into key/value pairs sequence file. (Cheng Zhang via Eric Yang)

    CHUKWA-55. Aggregate data from HDFS bytes usage, and Mapreduce Job slot time to compute user usage of the cluster. (Eric Yang)

    CHUKWA-50.  Added parser for extract Job History and Job Conf into key value pairs, 
                and database loader dictionary file. (Contribute by Cheng Zhang via Eric Yang)

    CHUKWA-69.  Calculate trapezoid area for a given series of data.  (Contribute by Cheng Zhang via Eric Yang)

    CHUKWA-62.  Add script to start tailing files.  (asrabkin)
    
    CHUKWA-12.  Add instrumention Api for Chukwa components.  (Jerome Boulon via asrabkin)

    CHUKWA-14.  Added permalink support to HICC GUI. (Eric Yang)

    CHUKWA-45.  Added docs to folder structure. (Corinne Chandel via asrabkin)

    CHUKWA-13.  Create a new Macro class for macro substitution.
                Changed Aggregator to use Macro class.
                Added macro substitution support to Chukwa Charting.
                Added Macro test case. (Eric Yang)

    HADOOP-4989. Added capability to add scatter chart. (Eric Yang)

  IMPROVEMENTS

    CHUKWA-226. Changed HDFS usage collection frequency from 10 minutes to 60 minutes. (Cheng Zhang via Eric Yang)

    CHUKWA-104. Remove permission setting code for Log4JMetricsContext and ChukwaDailyRollingFileAppender.  (Jerome Boulon via Eric Yang)

    CHUKWA-178. Extend aggregation by 5 minutes for cluster metrics. (Eric Yang)

    CHUKWA-176. Rearrange the parameter order for aggregator.sh. (Eric Yang)

    CHUKWA-173. Parameterize configuration, and enable substitution at build time. (Jerome Boulon via Eric Yang)

    CHUKWA-131. Added additional Mapred job/task metrics.  (Eric Yang)

    CHUKWA-145. Tuned hadoop parameters for demux. (Jerome Boulon via Eric Yang)

    CHUKWA-163. Updated reference to mdl.xml file. (Terence Kwan via Eric Yang)

    CHUKWA-157. Added javadoc target, api-xml, api-report, and change log 2 html. (Eric Yang)

    CHUKWA-141. Updated Chukwa Docs overview page for 0.1.2. (Corinne Chandel via Eric Yang)

    CHUKWA-138. Updated Chukwa Admin Guide. (Corinne Chandel via Eric Yang)

    CHUKWA-128. Added tools to compute aggregation in database. (Eric Yang)

    CHUKWA-112. Updated README file.  (Corinne Chandel via Eric Yang)

    CHUKWA-134. Add release audit target. (Giridharan Kesavan via Eric Yang)

    CHUKWA-80.  Extracted rpm spec file from build.xml file.  Fix start up script, and config script. (Eric Yang)

    CHUKWA-26.  * DemuxManager, ArchiveManager and PostProcessorManager are now a single daemon process each.
                * Each one working independently from others, as soon as something is available.
                * Start-data-processor is now using those new daemons instead of pocessSink.sh
                * Daily will process a daily compaction only when all hourly would have been done.
                * Demux is now able to send NSCA commands to Nagios. (Jerome Boulon via Eric Yang)

    CHUKWA-81.  Fix file descriptor leak.  (asrabkin)

    CHUKWA-29.  Added TaskTracker and DataNode client trace log parser and database loader.  (Chris Douglas via Eric Yang)

    CHUKWA-86. Improved JDBC compatibility layer (Cheng Zhang via Eric Yang)

    CHUKWA-64. Changed version number to 0.1.2, and release number to alpha. (Eric Yang)

    CHUKWA-59. Collect HDFS Usage information for users. (Contribute by Cheng Zhang via Eric Yang)

    CHUKWA-54. Filters web based input from HICC to prevent cross site scripting attack. (Eric Yang)

    CHUKWA-31. Added duration logging for SQL statements. (Eric Yang)
 
    HADOOP-5373. Collectors track lifetime-received chunks. (asrabkin)

    HADOOP-5370. Collectors don't write empty sink files (asrabkin)

    HADOOP-5228. Chukwa tests shouldn't write to /tmp. (asrabkin)

    HADOOP-5035. Improved Y axis ticker labelling.
                 Used TreeMap to build non-time series data for charting. 
                 Improved handling of "not a number "values. (Eric Yang)

    HADOOP-5030. Changed RPM install location to the value specified by build.properties file. (Eric Yang)

    HADOOP-5029. Added mdl script to manually load chukwa sequence file to database. (Eric Yang)

    HADOOP-4843. Enable job history log file streaming into Chukwa by using JobTrackerInstrumentation API. (Eric Yang)

    HADOOP-4827. * Replace Consolidator with Aggregator macros 
                 * Updated database table schema 
                 * Fix interval for aggregation execute 
                 * Improved SQL macros support, add group_avg() and past_*_minutes key word (Eric Yang)

  OPTIMIZATIONS

  BUG FIXES

    CHUKWA-249. Use ChukwaConfiguration in Demux. (Jerome Boulon via asrabkin)

    CHUKWA-257. fixed right arrow image for date picker. (Eric Yang)

    CHUKWA-251. Added per widget Time range input. (Terence Kwan via Eric Yang)
 
    CHUKWA-239. Demux settings now work out of the box. (Ari Rabkin)

    CHUKWA-250. Updated hadoop jar reference in chukwa config. (Cheng Zhang via Eric Yang)

    CHUKWA-211. Manage symlink correctly for RPM upgrade and uninstall. (Eric Yang)

    CHUKWA-245. Includes ChukwaJobTrackerInstrumentation class as part of the input tools compilation. (Eric Yang)

    CHUKWA-234. Improved handling of null session for XssFilter. (Eric Yang)

    CHUKWA-232. Fixed classpath for chukwa core jar file. (Eric Yang)

    CHUKWA-215. Corrected postProcess.sh environment setup. (Jerome Boulon via Eric Yang)

    CHUKWA-223. Renamed read only session caching object to cache.*, and skip those values for caching. (Eric Yang)

    CHUKWA-238.  Resolve race condition in archiving. (Ari Rabkin)

    CHUKWA-228.  Added rpm.hdfsusage.uid option to run HDFS usage as a separate user. (Cheng Zhang via Eric Yang)

    CHUKWA-227. Improved the user privilege for reading HDFS usage.  (Cheng Zhang via Eric Yang)

    CHUKWA-229. Fix descriptor leak for ExecPlugin. (Ari Rabkin via Eric Yang)

    CHUKWA-212. Fix file descriptor leak in MDL.  (Jerome Boulon via Eric Yang)

    CHUKWA-220. Correct min, max settings for yaxis charting. (Terence Kwan via Eric Yang)

    CHUKWA-222. Reduce Sar version dependency. (Ari Rabkin) 

    CHUKWA-103. Fix documentation broken link. (Eric Yang)

    CHUKWA-208. Skip copying of test/sample directory if it doesn't exist. (Eric Yang)

    CHUKWA-206. Removed hard coded path from configuration files, and sample data. (Eric Yang)

    CHUKWA-201. Removed sourcing of Timeline widget from simile.mit.edu. (Eric Yang)

    CHUKWA-193. Remove unrecognized tag from Job History Log file parser. (Cheng Zhang via Eric Yang)

    CHUKWA-189. Added JAVA_LIBRARY_PATH to chukwa-env.sh for enabling compression. (Jerome Boulon via Eric Yang)

    CHUKWA-186. Remove duplicated timestamp from aggregation script. (Eric Yang)

    CHUKWA-187. Correction to status for job and task status. (Cheng Zhang via Eric Yang)

    CHUKWA-183. Added HICC startup script. (Eric Yang)

    CHUKWA-177. Added test case for verify the value between JSON values and database values. (Terence Kwan via Eric Yang)

    CHUKWA-181. Changed chukwa check point file to $CHUKWA_LOG_DIR. (Jerome Boulon via Eric Yang)

    CHUKWA-182. Added nagios appender configuration. (Eric Yang)

    CHUKWA-158. Consolidate namenode address to a single configuration object. (Eric Yang)

    CHUKWA-168. Added watchdog for database. (Eric Yang)

    CHUKWA-175.  Removed error message for shutting down data processors. (Eric Yang)

    CHUKWA-155.  Store final job status only (Cheng Zhang via Eric Yang)

    CHUKWA-164.  Use year corresponding to the sender time stamp. (Cheng Zhang via Eric Yang)

    CHUKWA-166.  Handle null parameter case for XSSFilter. (Terence Kwan via Eric Yang)

    CHUKWA-156. Test Macro testcase changed to use timestamp check for the generated macros. (Eric Yang)

    CHUKWA-154.  Handle adaptor exception, close file pointers on failure condition.  (Jerome Boulon via Eric Yang)

    CHUKWA-139.  Rewrite collector bail out code.  (Cheng Zhang via Eric Yang)

    CHUKWA-119.  Removed dependency of ChukwaAgent from ChunkImpl for preventing multiple
                 MetricsContext to be initialized in the same VM. (Jerome Boulon via Eric Yang)


    CHUKWA-121.  Added logic to detect partition number less than 0.  (Eric Yang)

    CHUKWA-132.  Handle multi-line output in Job History file more gracefully. (Cheng Zhang via Eric Yang)

    CHUKWA-124.  Fixed JDBC connection initialization and closing.  Added escapeQuotes utility method. (Eric Yang)

    CHUKWA-122. Updated logic for finding time partition for Macro.  (Eric Yang)

    CHUKWA-125. Updated chukwa demux template to include JobHistory parser. (Eric Yang)

    CHUKWA-98.  Added Daemon watcher to capture signal for pid file management. (Cheng Zhang via Eric Yang)

    CHUKWA-120. Added the missing commons-cli library. (Jerome Boulon via Eric Yang)

    CHUKWA-61.  Added report widget for accounting information. (Eric Yang)

    CHUKWA-77.  Added Database Schema for Hadoop accounting information. (Eric Yang)

    CHUKWA-92.  AbstractMetricsContext was using the wrong value (Jerome Boulon via asrabkin)

    CHUKWA-70.  Rewrite FileAdaptor.  (Jerome Boulon via asrabkin)

    CHUKWA-93.  Fix NPE in SeqFileWriter.  (Jiaqi Tan via asrabkin)

    CHUKWA-1. Remove lzo job configuration from Chukwa data processors. (Contribute by Jerome Boulon via Eric Yang)

    CHUKWA-40. Check for null pointer exception before unregister adaptor. (Contribute By Jerome Boulon via Eric Yang)

    CHUKWA-75. Filter out iostat values that is greater than 1+e10.

    CHUKWA-47. ChukwaJobTrackerInstrumentation class extends JobTrackerMetricsInst and added 
               finalizedJob method for stop streaming job history log file. (Jerome Boulon via Eric Yang)

    CHUKWA-66. Corrected TerminatorThread logger class name. (Eric Yang)

    CHUKWA-65. Redirect metrics log file to CHUKWA_LOG_DIR, and setup read/writable permission on the metrics
               log files. (Eric Yang)


    CHUKWA-73. Added Socket Timeout 60 seconds. (Jerome Boulon via Eric Yang)

    CHUKWA-58. Changed watchdog to look for CHUKWA_PID_DIR. (Eric Yang)

    CHUKWA-43.  ChukwaLog4jAppender should send the current file offset instead of sending 0.  (Jerome Boulon via asrabkin)

    CHUKWA-41.  ChukwaLog4jAppender does not escape \n for exception.  (Jerome Boulon via asrabkin)

    CHUKWA-28.  Late initalization of log4j adaptor.  (Jerome Boulon via asrabkin)

    CHUKWA-48.  Cleanup code to resolve compiler warnings.  (asrabkin)

    CHUKWA-9.  MetricDataLoader should close JDBC connection.  (Jerome Boulon via asrabkin)

    CHUKWA-37.  Remove ChuckwaArchiveBuilder. (asrabkin)

    CHUKWA-33.  Reformat code to fit hadoop style.  (asrabkin)
 
    CHUKWA-8.  Remove deprecated conf files.  (eyang via asrabkin)

    CHUKWA-11.  Remove non .template files from conf dir. (asrabkin)

    HADOOP-5409. Changed opt directory copy to a optional operation, if opt exist. (Eric Yang)

    HADOOP-5138. Fixing failing test cases by separating TestCharFileTailing
    and TestFileTailing adaptors. (Jerome Boulon via acmurthy)

    HADOOP-5401. Standardize control port conf option name as chukwaAgent.control.port (asrabkin)

    HADOOP-5087. Fix regex for command parsing. (asrabkin)

    HADOOP-5057. Better test coverage for missing checkpoint cases. (asrabkin)

    HADOOP-5055. Changed alert.conf location from $CHUKWA_HOME/conf/alert.conf to $CHUKWA_CONF_DIR/alert.conf. (Eric Yang)

    HADOOP-5054. Improved database partitioning by date. (Eric Yang)

    HADOOP-5051. * Added macro token subsitution for sum(table_name) 
                 * Added correct hdfs throughput aggregation SQL macros. (Eric Yang)

    HADOOP-5032. Export CHUKWA_CONF_DIR in chukwa-config.sh.

    HADOOP-4960. Changed metrics time from current system time to data source time.

    HADOOP-4959. Support parsing of top output for Redhat EL 5.1. (Eric Yang)

    HADOOP-4916. * Added external property file to reference location and ownership of chukwa 
                 * Added ability to control the user name to run chukwa (Eric Yang)

    HADOOP-4914. Added description fields to chukwa init.d scripts.

    HADOOP-4889. Move chown from post install phase to build phase of the RPM file. (Eric Yang)

    HADOOP-4884. Change date format from dd/mm/yyyy to yyyy/mm/dd for display in the chart tool tips. (Eric Yang)

    HADOOP-4860. Changed test cases into 3 different test classes to prevent Agent from loading chukwa_check_point file which interfered with test cases. (Eric Yang)

    HADOOP-4825. Replaced jps with ps ax for shutdown scripts.
                 Clean up reference of jar file from top level instead of build directory. (Eric Yang)

    HADOOP-4805. Removed black list collector feature from Chukwa Agent HTTP Sender. (Eric Yang)

    HADOOP-4796. Change component unit test targets from "ant test-agent" to "ant -Dtestcase=TestAgent test". (Eric Yang)

    HADOOP-4791. Add build configuration parameter to specify where Chukwa will be installed for RPM packaging. (Eric Yang)


Release 0.1.1 - Unreleased

  IMPROVEMENTS

    HADOOP-4431. Add versionning/tags to Chukwa Chunk. 
    (Jerome Boulon via Johan)

    HADOOP-4433. Improve data loader for collecting metrics and log files.
    (Eric Yang via omalley)

    HADOOP-5205.  Change the value of CHUKWA_IDENT_STRING from demo to TODO-AGENTS-INSTANCE-NAME in chukwa-env.sh.template (Jerome Boulon via asrabkin)

    HADOOP-5033.  Simplified ChukwaWriter API.  (asrabkin)


  NEW FEATURES

    HADOOP-3719. Initial checkin of Chukwa, which is a data collection and 
    analysis framework. (Jerome Boulon, Andy Konwinski, Ari Rabkin, 
    and Eric Yang)