There are a number of options available to you for debugging Slider applications. They include:
There are a number of options for viewing the generated log files:
The log files are accessible via the Yarn Resource Manager UI. From the main page (e.g. http://${YARN_RESOURCE_MGR_HOST}:8088
), click on the link for the application instance of interest, and then click on the logs
link. This will present you with a page with links to the slider-err.txt
file and the slider-out.txt
file. The former is the file you should select -it is where the applicaton logs go Once the log page is presented, click on the link at the top of the page (“Click here for full log”) to view the entire file.
If the file slider-out.txt
is empty, then examine slider-err.txt
-an empty output log usually means that the java process failed to start -this should be logged in the error file.
If access to other log files is required, there is the option of logging in to the host machine on which the application component is running -provided you have the correct permissions.
The root directory for all YARN associated files is the value of yarn.nodemanager.log-dirs
in yarn-site.xml
- e.g. /hadoop/yarn/log
. Below the root directory you will find an application and container sub-directory (e.g. /application_1398372047522_0009/container_1398372047522_0009_01_000001/
). Below the container directory you will find any log files associated with the processes running in the given Yarn container.
Within a container log the following files are useful while debugging the application.
agent.log
E.g. application_1398098639743_0024/container_1398098639743_0024_01_000003/infra/log/agent.log
This file contains the logs from the Slider-Agent.
application component log
E.g. ./log/application_1398098639743_0024/container_1398098639743_0024_01_000003/app/log/hbase-yarn-regionserver-c6403.ambari.apache.org.log
The location of the application log is defined by the application. “${AGENT_LOG_ROOT}” is a symbol available to the app developers to use as a root folder for logging.
agent operations log
E.g. ./log/application_1398098639743_0024/container_1398098639743_0024_01_000003/app/command-log/
The command logs produced by the slider-agent are available in the command-log
folder relative to ${AGENT_LOG_ROOT}/app
Note that the fish shell is convenient for debugging, as cat log/**/slider-out.txt
will find the relevant output file irrespective of what the path leading to it is.
For situtations in which the logging does not yield enough information to debug an issue, the user has the option of specifying JVM command line options for the Application Master that enable attaching to the running process with a debugger (e.g. the remote debugging facilities in Eclipse or Intellij IDEA). In order to specify the JVM options, edit the application configuration file (the file specified as the --template
argument value on the command line for cluster creation) and specify the jvm.opts
property for the slider-appmaster
component:
`"components": { "slider-appmaster": { "jvm.heapsize": "256M", "jvm.opts": "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" }, ...`
You may specify suspend=y
in the line above if you wish to have the application master process wait for the debugger to attach before beginning its processing.