###Setup the HadoopMonitoring database
###Install Hadoop and SQLServerSink
Follow the hadoop install instructions here, making sure you download the 1.3 MSI.
Note the location of the clusterproperties.txt configuration file.
Setup Metrics2 SqlServerSink by editing the hadoop-metrics2.properties file in the hadoop bin folder (e.g. C:\HDP\hadoop-1.2.0.1.3.0.0-0380\bin\hadoop-metrics2.properties).
Note that this step should be repeated on each node of the cluster installed above. Set the following properties…
*.sink.sql.class=org.apache.hadoop.metrics2.sink.SqlServerSink namenode.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMonitoring;user=[user];password=[password] datanode.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMonitoring;user=[user];password=[password] jobtracker.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMonitoring;user=[user];password=[password] tasktracker.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMonitoring;user=[user];password=[password] maptask.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMonitoring;user=[user];password=[password] reducetask.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMonitoring;user=[user];password=[password]
Note that the server, port, user and password should work with the SQLServer install from above (e.g. jdbc:sqlserver://ambari1:1433;databaseName=HadoopMonitoring;user=sa;password=BigData1).
Copy the SqlServer JDBC driver jar and the ambari-scom jar to each node of the cluster.
Set ambari-scom jar and SQLServer driver jar in class path by editing the class path in the namenode.xml, datanode.xml, jobtracker.xml, tasktracker.xml and other component xml files of the hadoop bin folder (e.g. C:\HDP\hadoop-1.2.0.1.3.0.0-0380\bin\namenode xml).
Note that this step should be repeated on each node of the cluster. In each file prepend the following to -classpath…
[path to ambari-scom jar];[path to SqlServer JDBC jar];
For example, ‘-classpath C:\ambari-scom\target\ambari-scom-1.0.jar;C:\hadoop\sqljdbc4.jar;…’.
Start (or restart) Hadoop by using the scripts in the HDP install folder to start the services as directed by the link in step 1.
C:\HDP>start_local_hdp_services.cmd
Check to see that metrics are being directed to the SQLServer database by querying the records table.
select * from HadoopMonitoring.dbo.MetricRecord
The table should not be empty.
###Run AmbariServer
Edit the ambari.properties file from the ambari-scom project (ambari-scom/config/ambari.properties or unzip target/ambari-scom-1.0-conf.zip to desired location). Add the following properties ...
scom.sink.db.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver scom.sink.db.url=jdbc:sqlserver://[server]:[port];databaseName=HadoopMonitoring;user=[user];password=[password]
Note that the server, port, user and password should work with the SQLServer install from above (e.g. jdbc:sqlserver://ambari1:1433;databaseName=HadoopMonitoring;user=sa;password=BigData1).
Use java to run the ambari server (org.apache.ambari.scom.AmbariServer) from a Windows command prompt. Include the following in the classpath ...
The command should look something like this ...
java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -Xms512m -Xmx2048m -cp "c:\ambari-scom\conf;c:\hadoop\sqljdbc4.jar;c:\ambari-scom\target\ambari-scom-1.0.jar;c:\ambari-scom\target\lib\*" org.apache.ambari.scom.AmbariServer
###Test the API
From a browser access the API...
http://[server]:8080/api/v1/clusters
Verify that metrics are being reported…
http://[server]:8080/api/v1/clusters/ambari/services/HDFS/components/NAMENODE