docs/src/site/twiki/DG_QuickStart.twiki - oozie - Git at Google

 <noautolink>

 [[index][::Go back to Oozie Documentation Index::]]

 ---+!! Oozie Quick Start

 These instructions install and run Oozie using an embedded Tomcat server and an embedded Derby database.

 For detailed install and configuration instructions refer to [[AG_Install][Oozie Install]].

 %TOC%

 ---++ Building Oozie

 ---+++ System Requirements:
    * Unix box (tested on Mac OS X and Linux)
    * Java JDK 1.7+
    * Maven 3.0.1+
    * Hadoop 0.20.2+
    * Pig 0.7+

 JDK commands (java, javac) must be in the command path.

 The Maven command (mvn) must be in the command path.

 ---+++ Building Oozie

 Download a source distribution of Oozie from the "Releases" drop down menu on the [[http://oozie.apache.org][Oozie site]].

 Expand the source distribution =tar.gz= and change directories into it.

 The simplest way to build Oozie is to run the =mkdistro.sh= script:
 <verbatim>
 $ bin/mkdistro.sh [-DskipTests]

 Running =mkdistro.sh= will create the binary distribution of Oozie. By default, oozie war will not contain hadoop and
 hcatalog libraries, however they are required for oozie to work. There are 2 options to add these libraries:

 1. At install time, copy the hadoop and hcatalog libraries to libext and run oozie-setup.sh to setup oozie war. This is
 suitable when same oozie package needs to be used in multiple set-ups with different hadoop/hcatalog versions.

 2. Build with -Puber which will bundle the required libraries in the oozie war. Further, the following options are
 available to customise the versions of the dependencies:
 -P<profile> - default hadoop-2. Valid are hadoop-1, hadoop-2 or hadoop-3. Choose the correct hadoop
 profile depending on the hadoop version used.
 -Dhadoop.version=<version> - default 1.2.1 for hadoop-1, 2.4.0 for hadoop-2 and 3.0.0-SNAPSHOT for hadoop-3
 -Dhadoop.auth.version=<version> - defaults to hadoop version
 -Ddistcp.version=<version> - defaults to hadoop version
 -Dpig.version=<version> - default 0.16.0
 -Dpig.classifier=<classifier> - default none
 -Dsqoop.version=<version> - default 1.4.3
 -Dsqoop.classifier=<classifier> - default hadoop100
 -Dtomcat.version=<version> - default 6.0.53
 -Dopenjpa.version=<version> - default 2.2.2
 -Dxerces.version=<version> - default 2.10.0
 -Dcurator.version=<version> - default 2.5.0
 -Dhive.version=<version> - default 0.13.1 for hadoop-1, 1.2.0 for hadoop-2 and hadoop-3 profile
 -Dhbase.version=<version> - default 0.94.2
 </verbatim>

 More details on building Oozie can be found on the [[ENG_Building][Building Oozie]] page.

 ---++ Server Installation

 ---+++ System Requirements

    * Unix (tested in Linux and Mac OS X)
    * Java 1.7+
    * Hadoop
       * [[http://hadoop.apache.org][Apache Hadoop]] (tested with 1.2.1 & 2.4.0+)
    * ExtJS library (optional, to enable Oozie webconsole)
       * [[http://archive.cloudera.com/gplextras/misc/ext-2.2.zip][ExtJS 2.2]]

 The Java 1.7+ =bin= directory should be in the command path.

 ---+++ Server Installation

 *IMPORTANT:* Oozie ignores any set value for =OOZIE_HOME=, Oozie computes its home automatically.

    * Build an Oozie binary distribution
    * Download a Hadoop binary distribution
    * Download ExtJS library (it must be version 2.2)

 *NOTE:* The ExtJS library is not bundled with Oozie because it uses a different license.

 *NOTE:* Oozie UI browser compatibility Chrome (all), Firefox (3.5), Internet Explorer (8.0), Opera (10.5).

 *NOTE:* It is recommended to use a Oozie Unix user for the Oozie server.

 Expand the Oozie distribution =tar.gz=.

 Expand the Hadoop distribution =tar.gz= (as the Oozie Unix user).

 #HadoopProxyUser

 *NOTE:* Configure the Hadoop cluster with proxyuser for the Oozie process.

 The following two properties are required in Hadoop core-site.xml:

 <verbatim>
   <!-- OOZIE -->
   <property>
     <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>
     <value>[OOZIE_SERVER_HOSTNAME]</value>
   </property>
   <property>
     <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>
     <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>
   </property>
 </verbatim>

 Replace the capital letter sections with specific values and then restart Hadoop.

 The ExtJS library is optional (only required for the Oozie web-console to work)

 *IMPORTANT:* all Oozie server scripts (=oozie-setup.sh=, =oozied.sh=, =oozie-start.sh=, =oozie-run.sh=
 and =oozie-stop.sh=) run only under the Unix user that owns the Oozie installation directory,
 if necessary use =sudo -u OOZIE_USER= when invoking the scripts.

 As of Oozie 3.3.2, use of =oozie-start.sh=, =oozie-run.sh=, and =oozie-stop.sh= has
 been deprecated and will print a warning. The =oozied.sh= script should be used
 instead; passing it =start=, =run=, or =stop= as an argument will perform the
 behaviors of =oozie-start.sh=, =oozie-run.sh=, and =oozie-stop.sh= respectively.

 Create a *libext/* directory in the directory where Oozie was expanded.

 If using the ExtJS library copy the ZIP file to the *libext/* directory. If hadoop and hcatalog libraries are not
 already included in the war, add the corresponding libraries to *libext/* directory.

 A "sharelib create -fs fs_default_name [-locallib sharelib]" command is available when running oozie-setup.sh
 for uploading new sharelib into hdfs where the first argument is the default fs name
 and the second argument is the Oozie sharelib to install, it can be a tarball or the expanded version of it.
 If the second argument is omitted, the Oozie sharelib tarball from the Oozie installation directory will be used.
 Upgrade command is deprecated, one should use create command to create new version of sharelib.
 Sharelib files are copied to new lib_<timestamped> directory. At start, server picks the sharelib from latest time-stamp directory.
 While starting server also purge sharelib directory which is older than sharelib retention days
 (defined as oozie.service.ShareLibService.temp.sharelib.retention.days and 7 days is default).

 "prepare-war [-d directory]" command is for creating war files for oozie with an optional alternative directory other
 than libext.

 db create|upgrade|postupgrade -run [-sqlfile <FILE>] command is for create, upgrade or postupgrade oozie db with an
 optional sql file

 Run the =oozie-setup.sh= script to configure Oozie with all the components added to the *libext/* directory.

 <verbatim>
 $ bin/oozie-setup.sh prepare-war [-d directory] [-secure]
                      sharelib create -fs <FS_URI> [-locallib <PATH>]
                      sharelib upgrade -fs <FS_URI> [-locallib <PATH>]
                      db create|upgrade|postupgrade -run [-sqlfile <FILE>]
 </verbatim>

 The =-secure= option will configure Oozie to use HTTP (SSL); refer to
 [[AG_Install#Setting_Up_Oozie_with_HTTPS_SSL][Setting Up Oozie with HTTPS (SSL)]] for more details.

 Create the Oozie DB using the 'ooziedb.sh' command line tool:

 <verbatim>
 $ bin/ooziedb.sh create -sqlfile oozie.sql -run

 Validate DB Connection.
 DONE
 Check DB schema does not exist
 DONE
 Check OOZIE_SYS table does not exist
 DONE
 Create SQL schema
 DONE
 DONE
 Create OOZIE_SYS table
 DONE

 Oozie DB has been created for Oozie version '3.2.0'

 $
 </verbatim>

 Start Oozie as a daemon process run:

 <verbatim>
 $ bin/oozied.sh start
 </verbatim>

 To start Oozie as a foreground process run:

 <verbatim>
 $ bin/oozied.sh run
 </verbatim>

 Check the Oozie log file =logs/oozie.log= to ensure Oozie started properly.

 Using the Oozie command line tool check the status of Oozie:

 <verbatim>
 $ bin/oozie admin -oozie http://localhost:11000/oozie -status
 </verbatim>

 Using a browser go to the [[http://localhost:11000/oozie][Oozie web console]], Oozie status should be *NORMAL*.

 Refer to the [[DG_Examples][Running the Examples]] document for details on running the examples.

 ---++ Client Installation

 ---+++ System Requirements

    * Unix (tested in Linux and Mac OS X)
    * Java 1.7+

 The Java 1.7+ =bin= directory should be in the command path.

 ---+++ Client Installation

 Copy and expand the =oozie-client= TAR.GZ file bundled with the distribution. Add the =bin/= directory to the =PATH=.

 Refer to the [[DG_CommandLineTool][Command Line Interface Utilities]] document for a full reference of the =oozie=
 command line tool.

 NOTE: The Oozie server installation includes the Oozie client. The Oozie client should be installed in remote machines
 only.

 #OozieShareLib
 ---++ Oozie Share Lib Installation

 Oozie share lib has been installed by oozie-setup.sh create command explained in the earlier section.

 See the [[WorkflowFunctionalSpec#ShareLib][Workflow Functional Specification]] and [[AG_Install#Oozie_Share_Lib][Installation]] for more information about the Oozie ShareLib.

 [[index][::Go back to Oozie Documentation Index::]]

 </noautolink>
	<noautolink>

	[[index][::Go back to Oozie Documentation Index::]]

	---+!! Oozie Quick Start

	These instructions install and run Oozie using an embedded Tomcat server and an embedded Derby database.

	For detailed install and configuration instructions refer to [[AG_Install][Oozie Install]].

	%TOC%

	---++ Building Oozie

	---+++ System Requirements:
	* Unix box (tested on Mac OS X and Linux)
	* Java JDK 1.7+
	* Maven 3.0.1+
	* Hadoop 0.20.2+
	* Pig 0.7+

	JDK commands (java, javac) must be in the command path.

	The Maven command (mvn) must be in the command path.

	---+++ Building Oozie

	Download a source distribution of Oozie from the "Releases" drop down menu on the [[http://oozie.apache.org][Oozie site]].

	Expand the source distribution =tar.gz= and change directories into it.

	The simplest way to build Oozie is to run the =mkdistro.sh= script:
	<verbatim>
	$ bin/mkdistro.sh [-DskipTests]

	Running =mkdistro.sh= will create the binary distribution of Oozie. By default, oozie war will not contain hadoop and
	hcatalog libraries, however they are required for oozie to work. There are 2 options to add these libraries:

	1. At install time, copy the hadoop and hcatalog libraries to libext and run oozie-setup.sh to setup oozie war. This is
	suitable when same oozie package needs to be used in multiple set-ups with different hadoop/hcatalog versions.

	2. Build with -Puber which will bundle the required libraries in the oozie war. Further, the following options are
	available to customise the versions of the dependencies:
	-P<profile> - default hadoop-2. Valid are hadoop-1, hadoop-2 or hadoop-3. Choose the correct hadoop
	profile depending on the hadoop version used.
	-Dhadoop.version=<version> - default 1.2.1 for hadoop-1, 2.4.0 for hadoop-2 and 3.0.0-SNAPSHOT for hadoop-3
	-Dhadoop.auth.version=<version> - defaults to hadoop version
	-Ddistcp.version=<version> - defaults to hadoop version
	-Dpig.version=<version> - default 0.16.0
	-Dpig.classifier=<classifier> - default none
	-Dsqoop.version=<version> - default 1.4.3
	-Dsqoop.classifier=<classifier> - default hadoop100
	-Dtomcat.version=<version> - default 6.0.53
	-Dopenjpa.version=<version> - default 2.2.2
	-Dxerces.version=<version> - default 2.10.0
	-Dcurator.version=<version> - default 2.5.0
	-Dhive.version=<version> - default 0.13.1 for hadoop-1, 1.2.0 for hadoop-2 and hadoop-3 profile
	-Dhbase.version=<version> - default 0.94.2
	</verbatim>

	More details on building Oozie can be found on the [[ENG_Building][Building Oozie]] page.

	---++ Server Installation

	---+++ System Requirements

	* Unix (tested in Linux and Mac OS X)
	* Java 1.7+
	* Hadoop
	* [[http://hadoop.apache.org][Apache Hadoop]] (tested with 1.2.1 & 2.4.0+)
	* ExtJS library (optional, to enable Oozie webconsole)
	* [[http://archive.cloudera.com/gplextras/misc/ext-2.2.zip][ExtJS 2.2]]

	The Java 1.7+ =bin= directory should be in the command path.

	---+++ Server Installation

	IMPORTANT: Oozie ignores any set value for =OOZIE_HOME=, Oozie computes its home automatically.

	* Build an Oozie binary distribution
	* Download a Hadoop binary distribution
	* Download ExtJS library (it must be version 2.2)

	NOTE: The ExtJS library is not bundled with Oozie because it uses a different license.

	NOTE: Oozie UI browser compatibility Chrome (all), Firefox (3.5), Internet Explorer (8.0), Opera (10.5).

	NOTE: It is recommended to use a Oozie Unix user for the Oozie server.

	Expand the Oozie distribution =tar.gz=.

	Expand the Hadoop distribution =tar.gz= (as the Oozie Unix user).

	#HadoopProxyUser

	NOTE: Configure the Hadoop cluster with proxyuser for the Oozie process.

	The following two properties are required in Hadoop core-site.xml:

	<verbatim>
	<!-- OOZIE -->
	<property>
	<name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>
	<value>[OOZIE_SERVER_HOSTNAME]</value>
	</property>
	<property>
	<name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>
	<value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>
	</property>
	</verbatim>

	Replace the capital letter sections with specific values and then restart Hadoop.

	The ExtJS library is optional (only required for the Oozie web-console to work)

	IMPORTANT: all Oozie server scripts (=oozie-setup.sh=, =oozied.sh=, =oozie-start.sh=, =oozie-run.sh=
	and =oozie-stop.sh=) run only under the Unix user that owns the Oozie installation directory,
	if necessary use =sudo -u OOZIE_USER= when invoking the scripts.

	As of Oozie 3.3.2, use of =oozie-start.sh=, =oozie-run.sh=, and =oozie-stop.sh= has
	been deprecated and will print a warning. The =oozied.sh= script should be used
	instead; passing it =start=, =run=, or =stop= as an argument will perform the
	behaviors of =oozie-start.sh=, =oozie-run.sh=, and =oozie-stop.sh= respectively.

	Create a libext/ directory in the directory where Oozie was expanded.

	If using the ExtJS library copy the ZIP file to the libext/ directory. If hadoop and hcatalog libraries are not
	already included in the war, add the corresponding libraries to libext/ directory.

	A "sharelib create -fs fs_default_name [-locallib sharelib]" command is available when running oozie-setup.sh
	for uploading new sharelib into hdfs where the first argument is the default fs name
	and the second argument is the Oozie sharelib to install, it can be a tarball or the expanded version of it.
	If the second argument is omitted, the Oozie sharelib tarball from the Oozie installation directory will be used.
	Upgrade command is deprecated, one should use create command to create new version of sharelib.
	Sharelib files are copied to new lib_<timestamped> directory. At start, server picks the sharelib from latest time-stamp directory.
	While starting server also purge sharelib directory which is older than sharelib retention days
	(defined as oozie.service.ShareLibService.temp.sharelib.retention.days and 7 days is default).

	"prepare-war [-d directory]" command is for creating war files for oozie with an optional alternative directory other
	than libext.

	db create\|upgrade\|postupgrade -run [-sqlfile <FILE>] command is for create, upgrade or postupgrade oozie db with an
	optional sql file

	Run the =oozie-setup.sh= script to configure Oozie with all the components added to the libext/ directory.

	<verbatim>
	$ bin/oozie-setup.sh prepare-war [-d directory] [-secure]
	sharelib create -fs <FS_URI> [-locallib <PATH>]
	sharelib upgrade -fs <FS_URI> [-locallib <PATH>]
	db create\|upgrade\|postupgrade -run [-sqlfile <FILE>]
	</verbatim>

	The =-secure= option will configure Oozie to use HTTP (SSL); refer to
	[[AG_Install#Setting_Up_Oozie_with_HTTPS_SSL][Setting Up Oozie with HTTPS (SSL)]] for more details.

	Create the Oozie DB using the 'ooziedb.sh' command line tool:

	<verbatim>
	$ bin/ooziedb.sh create -sqlfile oozie.sql -run

	Validate DB Connection.
	DONE
	Check DB schema does not exist
	DONE
	Check OOZIE_SYS table does not exist
	DONE
	Create SQL schema
	DONE
	DONE
	Create OOZIE_SYS table
	DONE

	Oozie DB has been created for Oozie version '3.2.0'

	$
	</verbatim>

	Start Oozie as a daemon process run:

	<verbatim>
	$ bin/oozied.sh start
	</verbatim>

	To start Oozie as a foreground process run:

	<verbatim>
	$ bin/oozied.sh run
	</verbatim>

	Check the Oozie log file =logs/oozie.log= to ensure Oozie started properly.

	Using the Oozie command line tool check the status of Oozie:

	<verbatim>
	$ bin/oozie admin -oozie http://localhost:11000/oozie -status
	</verbatim>

	Using a browser go to the [[http://localhost:11000/oozie][Oozie web console]], Oozie status should be NORMAL.

	Refer to the [[DG_Examples][Running the Examples]] document for details on running the examples.

	---++ Client Installation

	---+++ System Requirements

	* Unix (tested in Linux and Mac OS X)
	* Java 1.7+

	The Java 1.7+ =bin= directory should be in the command path.

	---+++ Client Installation

	Copy and expand the =oozie-client= TAR.GZ file bundled with the distribution. Add the =bin/= directory to the =PATH=.

	Refer to the [[DG_CommandLineTool][Command Line Interface Utilities]] document for a full reference of the =oozie=
	command line tool.

	NOTE: The Oozie server installation includes the Oozie client. The Oozie client should be installed in remote machines
	only.

	#OozieShareLib
	---++ Oozie Share Lib Installation

	Oozie share lib has been installed by oozie-setup.sh create command explained in the earlier section.

	See the [[WorkflowFunctionalSpec#ShareLib][Workflow Functional Specification]] and [[AG_Install#Oozie_Share_Lib][Installation]] for more information about the Oozie ShareLib.

	[[index][::Go back to Oozie Documentation Index::]]

	</noautolink>