DEVNOTES - logging-flume - Git at Google

 Flume Developer Notes
 =====================

 // This is in asciidoc markup

 == Introduction

 This is meant to be a a guide for issues that occur when building,
 debugging and setting up Flume as developer.

 == High level directory and file structure.

 Flume uses the Maven build system and has a Maven project object model
 (pom) that has many components broken down into Maven modules.  Below
 we describe the contents of different directories.

 ----
 ./bin/                  Flume startup scripts
 ./conf/                 Flume configuration file samples
 ./flume-ng-core         Flume core module
 ./flume-ng-dist         Flume distribution package module
 ./flume-ng-channels     Flume channels modules (memory, JDBC and File)
 ./flume-ng-node         Flume node module
 ./flume-ng-sinks        Flume sink modules (HDFS)
 ----

 The files exclusions in `.gitignore` are either autogenerated by Maven or Eclipse.

 == Building and Testing Flume


 === Prerequisites

 You need to have Apache Maven 3.x installed.

 === Using Maven

 We are using Maven v3.x.  The Maven build system steps through several phases
 to create build artefacts.  At the highest level, the phases that are relevent
 to most devs are "compile" -> "test" -> "package" -> "install".

 Set MAVEN_OPTS to give the Flume build enough RAM to build.

   export MAVEN_OPTS="-Xms512m -Xmx1024m"

 Builds
 ------

 A development build that runs unit tests and installs to local Maven repo.
 This builds and tests all plugins.

 ----
 mvn install
 ----

 A development build that skips the execution of unit tests.

 ----
 mvn install -DskipTests
 ----

 A development build that runs unit tests. (no package generation)

 ----
 mvn test
 ----

 A development build that runs unit tests including only specific tests
 (where <TestFile> is a regex of a class name without .java or .class
 or path).

 ----
 mvn test -Dtest=<ClassRegex>
 ----

 ==== Including or excluding specific sets of tests.

 We've added hooks to the maven build that will enable you to exclude
 or include specific tests on a test run.  This is useful for excluding
 flakey tests or making a build that focuses solely upon flakey tests.

 To do this we created two variables:

 # test.include.pattern
 # test.exclude.pattern

 These variables take regular expression patterns of the files to be
 included or excluded.

 For the next set of examples, let's say you have flakey test called
 TestFlaky1 and TestFlaky2.

 You can execute tests that skip TestFlaky1 and TestFlaky2 by using the
 following command line:

 ----
 mvn test -Dtest.exclude.pattern=**/TestFlaky*.java
 ----

 Alternately, you could be more explicit

 ----
 mvn test -Dtest.exclude.pattern=**/TestFlaky1.java,**/TestFlaky2.java
 ----

 Conversely, you could execute only the flaky tests by using:

 ----
 mvn test -Dtest.include.pattern=**/TestFlaky*.java
 ----

 You can also have a combination of imports and exports.  This runs
 TestFlaky* but skips over TestFlaky2:

 ----
 mvn test -Dtest.include.pattern=**/TestFlaky*.java -Dtest.exclude.pattern=**/TestFlaky2.java
 ----

 NOTE: Both test.exclude.pattern and test.include.pattern get
 overridden if the test parameter is used.  Consider:

 ----
 mvn test -Dtest.exclude.pattern=**/TestFlaky*.java -Dtest=TestFlaky1
 ---

 In this case, TestFlaky1 will be run despite being in the
 test.exclude.pattern.

 === Running the most recent build

 To run the most recent build of Flume, first build the distribuion
 packages.

 ----
 mvn install -DskipTests
 ----

 == Integrated Development Environments for Flume

 Currently most Flume developers use the Eclipse IDE. We have included
 some instructions for getting started with Eclipse.

 === Setting up a Flume Eclipse projects from the Maven POMs.

 If you use Eclipse we suggest you use the m2eclipse plugin available
 here to properly create an environment for dev and testing in Eclipse.

 http://m2eclipse.sonatype.org/

 After installing it in Eclipse you will want to "Import" the Flume
 pom.xml project.

 This can be done by going to the Eclipse applications menu, navigating
 to File > Import... > Existing Maven Projects.  From there, browse to
 and select the directory that contains the root of the Flume project.

 == Rules of the Repository

 We have a few basic rules for code in the repository.

 The master/trunk pointer:

 * MUST always build.
 * SHOULD always pass all unit tests

 When commitng code we tag pushes with JIRA numbers, and their short descriptions.
 Generally these are in the following format:

 ----
 FLUME-42: Description from the jira
 ----

 All source files must include the following header (or a variant
 depending on comment characters):

 ----
 /**
  *      Licensed to the Apache Software Foundation (ASF) under one
  *      or more contributor license agreements.  See the NOTICE file
  *      distributed with this work for additional information
  *      regarding copyright ownership.  The ASF licenses this file
  *      to you under the Apache License, Version 2.0 (the
  *      "License"); you may not use this file except in compliance
  *      with the License.  You may obtain a copy of the License at
  *
  *        http://www.apache.org/licenses/LICENSE-2.0
  *
  *      Unless required by applicable law or agreed to in writing,
  *      software distributed under the License is distributed on an
  *      "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  *      KIND, either express or implied.  See the License for the
  *      specific language governing permissions and limitations
  *      under the License.
  */
 ----

 No build generated files should be checked in.  Here are some examples
 of generate files that should not be checked:

 * html documentation
 * avro-generated source
 * auto-generated versioning annotations
	Flume Developer Notes
	=====================

	// This is in asciidoc markup

	== Introduction

	This is meant to be a a guide for issues that occur when building,
	debugging and setting up Flume as developer.

	== High level directory and file structure.

	Flume uses the Maven build system and has a Maven project object model
	(pom) that has many components broken down into Maven modules. Below
	we describe the contents of different directories.

	----
	./bin/ Flume startup scripts
	./conf/ Flume configuration file samples
	./flume-ng-core Flume core module
	./flume-ng-dist Flume distribution package module
	./flume-ng-channels Flume channels modules (memory, JDBC and File)
	./flume-ng-node Flume node module
	./flume-ng-sinks Flume sink modules (HDFS)
	----

	The files exclusions in `.gitignore` are either autogenerated by Maven or Eclipse.

	== Building and Testing Flume


	=== Prerequisites

	You need to have Apache Maven 3.x installed.

	=== Using Maven

	We are using Maven v3.x. The Maven build system steps through several phases
	to create build artefacts. At the highest level, the phases that are relevent
	to most devs are "compile" -> "test" -> "package" -> "install".

	Set MAVEN_OPTS to give the Flume build enough RAM to build.

	export MAVEN_OPTS="-Xms512m -Xmx1024m"

	Builds
	------

	A development build that runs unit tests and installs to local Maven repo.
	This builds and tests all plugins.

	----
	mvn install
	----

	A development build that skips the execution of unit tests.

	----
	mvn install -DskipTests
	----

	A development build that runs unit tests. (no package generation)

	----
	mvn test
	----

	A development build that runs unit tests including only specific tests
	(where <TestFile> is a regex of a class name without .java or .class
	or path).

	----
	mvn test -Dtest=<ClassRegex>
	----

	==== Including or excluding specific sets of tests.

	We've added hooks to the maven build that will enable you to exclude
	or include specific tests on a test run. This is useful for excluding
	flakey tests or making a build that focuses solely upon flakey tests.

	To do this we created two variables:

	# test.include.pattern
	# test.exclude.pattern

	These variables take regular expression patterns of the files to be
	included or excluded.

	For the next set of examples, let's say you have flakey test called
	TestFlaky1 and TestFlaky2.

	You can execute tests that skip TestFlaky1 and TestFlaky2 by using the
	following command line:

	----
	mvn test -Dtest.exclude.pattern=*/TestFlaky.java
	----

	Alternately, you could be more explicit

	----
	mvn test -Dtest.exclude.pattern=/TestFlaky1.java,/TestFlaky2.java
	----

	Conversely, you could execute only the flaky tests by using:

	----
	mvn test -Dtest.include.pattern=*/TestFlaky.java
	----

	You can also have a combination of imports and exports. This runs
	TestFlaky* but skips over TestFlaky2:

	----
	mvn test -Dtest.include.pattern=*/TestFlaky.java -Dtest.exclude.pattern=**/TestFlaky2.java
	----

	NOTE: Both test.exclude.pattern and test.include.pattern get
	overridden if the test parameter is used. Consider:

	----
	mvn test -Dtest.exclude.pattern=*/TestFlaky.java -Dtest=TestFlaky1
	---

	In this case, TestFlaky1 will be run despite being in the
	test.exclude.pattern.

	=== Running the most recent build

	To run the most recent build of Flume, first build the distribuion
	packages.

	----
	mvn install -DskipTests
	----

	== Integrated Development Environments for Flume

	Currently most Flume developers use the Eclipse IDE. We have included
	some instructions for getting started with Eclipse.

	=== Setting up a Flume Eclipse projects from the Maven POMs.

	If you use Eclipse we suggest you use the m2eclipse plugin available
	here to properly create an environment for dev and testing in Eclipse.

	http://m2eclipse.sonatype.org/

	After installing it in Eclipse you will want to "Import" the Flume
	pom.xml project.

	This can be done by going to the Eclipse applications menu, navigating
	to File > Import... > Existing Maven Projects. From there, browse to
	and select the directory that contains the root of the Flume project.

	== Rules of the Repository

	We have a few basic rules for code in the repository.

	The master/trunk pointer:

	* MUST always build.
	* SHOULD always pass all unit tests

	When commitng code we tag pushes with JIRA numbers, and their short descriptions.
	Generally these are in the following format:

	----
	FLUME-42: Description from the jira
	----

	All source files must include the following header (or a variant
	depending on comment characters):

	----
	/**
	* Licensed to the Apache Software Foundation (ASF) under one
	* or more contributor license agreements. See the NOTICE file
	* distributed with this work for additional information
	* regarding copyright ownership. The ASF licenses this file
	* to you under the Apache License, Version 2.0 (the
	* "License"); you may not use this file except in compliance
	* with the License. You may obtain a copy of the License at
	*
	* http://www.apache.org/licenses/LICENSE-2.0
	*
	* Unless required by applicable law or agreed to in writing,
	* software distributed under the License is distributed on an
	* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	* KIND, either express or implied. See the License for the
	* specific language governing permissions and limitations
	* under the License.
	*/
	----

	No build generated files should be checked in. Here are some examples
	of generate files that should not be checked:

	* html documentation
	* avro-generated source
	* auto-generated versioning annotations