Knox provides gateway functionality for the REST APIs of the ResourceManager. The ResourceManager REST APIs allow the user to get information about the cluster - status on the cluster, metrics on the cluster, scheduler information, information about nodes in the cluster, and information about applications on the cluster. Also as of Hadoop version 2.5.0, the user can submit a new application as well as kill it (or get state) using the ‘Writable’ APIs.
The docs for this can be found here
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
To enable this functionality, a topology file needs to have the following configuration:
<service>
<role>RESOURCEMANAGER</role>
<url>http://<hostname>:<port>/ws</url>
</service>
The default resource manager http port is 8088. If it is configured to some other port, that configuration can be found in yarn-site.xml under the property yarn.resourcemanager.webapp.address.
For Yarn URLs, the mapping of Knox Gateway accessible URLs to direct Yarn URLs is the following.
| ------- | ------------------------------------------------------------------------------------- | | Gateway | https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/resourcemanager | | Cluster | http://{yarn-host}:{yarn-port}/ws} |
Some of the various calls that can be made and examples using curl are listed below.
# 0. Getting cluster info
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster'
# 1. Getting cluster metrics
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/metrics'
To get the same information in an xml format
curl -ikv -u guest:guest-password -H Accept:application/xml -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/metrics'
# 2. Getting scheduler information
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/scheduler'
# 3. Getting all the applications listed and their information
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps'
# 4. Getting applications statistics
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/appstatistics'
Also query params can be used as below to filter the results
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/appstatistics?states=accepted,running,finished&applicationTypes=mapreduce'
# 5. To get a specific application (please note, replace the application id with a real value)
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/{application_id}'
# 6. To get the attempts made for a particular application
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/{application_id}/appattempts'
# 7. To get information about the various nodes
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/nodes'
Also to get a specific node, use an id obtained in the response from above (the node id is scrambled) and issue the following
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/nodes/{node_id}'
# 8. To create a new Application
curl -ikv -u guest:guest-password -X POST 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/new-application'
An application id is returned from the request above and this can be used to submit an application.
# 9. To submit an application, put together a request containing the application id received in the above response (please refer to Yarn REST
API documentation).
curl -ikv -u guest:guest-password -T request.json -H Content-Type:application/json -X POST 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps'
Here the request is saved in a file called request.json
#10. To get application state
curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/{application_id}/state'
curl -ikv -u guest:guest-password -H Content-Type:application/json -X PUT -T state-killed.json 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/application_1409008107556_0007/state'
# 11. To kill an application that is running issue the below command with the application id of the application that is to be killed.
The contents of the state-killed.json file are :
{
"state":"KILLED"
}
curl -ikv -u guest:guest-password -H Content-Type:application/json -X PUT -T state-killed.json 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/{application_id}/state'
Note: The sensitive data like nodeHttpAddress, nodeHostName, id will be hidden.