To scale a cluster up or down, use the Cluster API. The Cluster API provide flexup and flex down capability the changes the size of one or more instances in a cluster. The instance size is a profile parameter that is a predefined value of zero, small, medium, and large. These predefined values are specified in the Myriad configuration file (myriad-config-default.yml). To specify the number of instances for a service, use the Service API. Services are configured in the Myriad configuration file (myriad-config-default.yml). To retrieve the Myriad configuration and the Myriad Scheduler state, use the Configuration API and State API.
The Myriad REST API provides the following functionality:
API | HTTP Method | URI | Description |
---|---|---|---|
Cluster | PUT | /api/cluster/flexup | Expands the cluster size |
Cluster | PUT | /api/cluster/flexdown | Shrinks the cluster size |
Service | PUT | /api/cluster/flexupservice | Increases the number of instances for a service. |
Service | PUT | /api/cluster/flexdownservice | Shrinks the number of instances for a service. |
Configuration | GET | /api/config | Retrieves the Myriad configuration. |
State | GET | /api/state | Retrieves a snapshot of the Myriad Scheduler state. |
Framework Shutdown | GET | /api/framework/shutdown/framework | Shuts down Myriad framework. |
The Cluster REST API uses the PUT /api/cluster/flexup and flexdown HTTP method and URI to expand and shrink the cluster size.
PUT /api/cluster/flexup // Expands the size of the YARN cluster. PUT /api/cluster/flexdown // Shrinks the size of the YARN cluster.
Parameters include:
Parameter | Description |
---|---|
profile | (Required) If a profile value is not specified, the API returns an error. The profile indicates the amount of resources (CPU or memory) a Node Manager should advertise to the Resource Manager. Default profiles: zero, small, medium, large. These default profiles (zero, small, medium, and large) are defined in the myriad-config-default.yml file. The resources associated with these default profiles can be modified; additionally, new profiles can be defined. |
instances | (Required) The number of Node Managers instances to launch. Each Node Manager instance advertises the amount of resources specified in the profile. The value is a number in the range of zero (0) to the number of Mesos slave nodes. |
constraints | (Optional) Array definition for a single constraint using the LIKE operator constraint format: <mesos_slave_attribute |
<resource_manager_host>:8192/api/cluster/flexup profile=<zero|small|medium|large> instances=<integer> constraints=<["JSON array of strings"]> <resource_manager_host>:8192/api/cluster/flexdown profile=<zero|small|medium|large> instances=<integer> constraints=<["JSON array of strings"]
Curl request example to flexup two instances with the profile set to small:
curl -X PUT http://10.10.100.19:8192/api/cluster/flexup -d instances=2 -d profile=small
Curl request example to flexdown one instance with the profile set to small:
curl -X PUT http://10.10.100.19:8192/api/cluster/flexdown -d instances=1 -d profile=small
Curl request example to launch two (2) Node Managers with profile set to large only on specific hosts, host-120 through host-129:
curl -X PUT http://10.10.100.19:8192/api/cluster/flexdown -d instances=2 -d profile=large -d constraints=["hostname LIKE host-12[0-9].example.com"]
Request header to flexup:
PUT /api/cluster/flexup HTTP/1.1 Host: 10.10.100.19:8192 Connection: keep-alive Content-Length: 36 Origin: http://10.10.100.19:8192 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 Content-Type: application/json Accept: */* Referer: http://10.10.100.19:8192/ Accept-Encoding: gzip, deflate, sdch Accept-Language: en-US,en;q=0.8
Request header to flexdown:
PUT /api/cluster/flexdown HTTP/1.1 Host: 10.10.100.19:8192 Connection: keep-alive Content-Length: 17 Origin: http://10.10.100.19:8192 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 Content-Type: application/json Accept: */* Referer: http://10.10.100.19:8192/ Accept-Encoding: gzip, deflate, sdch Accept-Language: en-US,en;q=0.8
Launches a Node Manager with the profile set to medium on any host in the Mesos cluster:
{profile: "medium", instances:1}
Launches a Node Manager with the profile set to small on any host in the Mesos cluster:
{ "instances":1, "profile": "small" }
Launches a Node Manager with the profile set to zero on any host in the Mesos cluster:
{ "instances":1 }
Launches four (4) Node Managers with profile set to large only on specific hosts, host-120 through host-129:
{ "instances":4, "profile": "large", "constraints": ["hostname LIKE host-12[0-9].example.com"] }
Launches two (2) Node Managers with profile set to zero only on hosts sharing a common Mesos slave attribute (http://mesos.apache.org/documentation/attributes-resources):
{ "instances":2, "profile": "zero", "constraints": ["hdfs LIKE true"] }
202 ACCEPTED
The Cluster REST API uses the PUT /api/cluster/flexup and flexdown HTTP method and URI to expand and shrink the cluster size.
PUT /api/cluster/flexup // Expands the size of the YARN cluster. PUT /api/cluster/flexdown // Shrinks the size of the YARN cluster.
Parameters include:
Parameter | Description |
---|---|
profile | (Required) If a profile value is not specified, the API returns an error. The profile indicates the amount of resources (CPU or memory) a Node Manager should advertise to the Resource Manager. Default profiles: zero, small, medium, large. These default profiles (zero, small, medium, and large) are defined in the myriad-config-default.yml file. The resources associated with these default profiles can be modified; additionally, new profiles can be defined. |
instances | (Required) The number of Node Managers instances to launch. Each Node Manager instance advertises the amount of resources specified in the profile. The value is a number in the range of zero (0) to the number of Mesos slave nodes. |
constraints | (Optional) Array definition for a single constraint using the LIKE operator constraint format: <mesos_slave_attribute |
<resource_manager_host>:8192/api/cluster/flexup profile=<zero|small|medium|large> instances=<integer> constraints=<["JSON array of strings"]> <resource_manager_host>:8192/api/cluster/flexdown profile=<zero|small|medium|large> instances=<integer> constraints=<["JSON array of strings"]>
Curl request example to flexup two instances with the profile set to small:
curl -X PUT http://10.10.100.19:8192/api/cluster/flexup -d instances=2 -d profile=small
Curl request example to flexdown one instance with the profile set to small:
curl -X PUT http://10.10.100.19:8192/api/cluster/flexdown -d instances=1 -d profile=small
Curl request example to launch two (2) Node Managers with profile set to large only on specific hosts, host-120 through host-129:
curl -X PUT http://10.10.100.19:8192/api/cluster/flexdown -d instances=2 -d profile=large -d constraints=["hostname LIKE host-12[0-9].example.com"]
Request header to flexup:
PUT /api/cluster/flexup HTTP/1.1 Host: 10.10.100.19:8192 Connection: keep-alive Content-Length: 36 Origin: http://10.10.100.19:8192 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 Content-Type: application/json Accept: */* Referer: http://10.10.100.19:8192/ Accept-Encoding: gzip, deflate, sdch Accept-Language: en-US,en;q=0.8
Request header to flexdown:
PUT /api/cluster/flexdown HTTP/1.1 Host: 10.10.100.19:8192 Connection: keep-alive Content-Length: 17 Origin: http://10.10.100.19:8192 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 Content-Type: application/json Accept: */* Referer: http://10.10.100.19:8192/ Accept-Encoding: gzip, deflate, sdch Accept-Language: en-US,en;q=0.8
Launches a Node Manager with the profile set to medium on any host in the Mesos cluster:
{profile: "medium", instances:1}
Launches a Node Manager with the profile set to small on any host in the Mesos cluster:
{ "instances":1, "profile": "small" }
Launches a Node Manager with the profile set to zero on any host in the Mesos cluster:
{ "instances":1 }
Launches four (4) Node Managers with profile set to large only on specific hosts, host-120 through host-129:
{ "instances":4, "profile": "large", "constraints": ["hostname LIKE host-12[0-9].example.com"] }
Launches two (2) Node Managers with profile set to zero only on hosts sharing a common Mesos slave attribute (http://mesos.apache.org/documentation/attributes-resources):
{ "instances":2, "profile": "zero", "constraints": ["hdfs LIKE true"] }
202 ACCEPTED
The State REST API uses the GET /api/config HTTP method and URI to retrieve configuration information.
GET /api/config
<resource_manager_host>:<port>/api/config
URL request example:
http://10.10.100.19:8192/api/config
Curl request example:
curl http://10.10.100.19:8192/api/config | python -m json.tool
Request header:
GET /api/config HTTP/1.1 Host: 10.10.100.19:8192 Connection: keep-alive Cache-Control: max-age=0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 Accept-Encoding: gzip, deflate, sdch Accept-Language: en-US,en;q=0.8
{ "checkpoint": false, "frameworkFailoverTimeout": 43200000.0, "frameworkName": "MyriadAlpha", "frameworkRole": "", "frameworkSuperUser": { "present": true }, "frameworkUser": { "present": true }, "mesosAuthenticationPrincipal": "", "mesosAuthenticationSecretFilename": "", "mesosMaster": "zk://10.10.100.19:2181/mesos", "myriadExecutorConfiguration": { "jvmMaxMemoryMB": { "present": true }, "nodeManagerUri": { "present": false }, "nativeLibrary": "/usr/local/lib/libmesos.so", "nmInstances": { "medium": 1 }, "nodeManagerConfiguration": { "cgroups": { "present": true }, "cpus": { "present": true }, "jvmMaxMemoryMB": { "present": true }, "jvmOpts": { "present": false } } "profiles": { "large": { "cpu": "10", "mem": "12288" }, "medium": { "cpu": "4", "mem": "4096" }, "small": { "cpu": "2", "mem": "2048" }, "zero": { "cpu": "0", "mem": "0" } }, "rebalancer": false, "haEnabled": false "restApiPort": 8192, "yarnEnvironment": { "YARN_HOME": "/root/hadoop-2.7.0" }, "zkServers": "10.10.100.19:2181", "zkTimeout": 20000 }
The State REST API uses the GET /api/state HTTP method and URI to retrieve a snapshot of the Myriad Scheduler state including pending, staging, active, and killable tasks.
GET /api/state
<resource_manager_host>:8192/api/state
URL request example:
http://10.10.100.19:8192/api/state
Curl request example:
curl http://10.10.100.19:8192/api/state | python -m json.tool
Request header:
GET /api/state HTTP/1.1 Host: 10.10.100.19:8192 Connection: keep-alive Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 Accept-Encoding: gzip, deflate, sdch Accept-Language: en-US,en;q=0.8
{ "activeTasks": [], "killableTasks": [], "pendingTasks": [ "nm.medium.ea564a5b-3a77-47dc-a7fe-2ff88ae2d5ed" ], "stagingTasks": [] }
The Framework REST API uses the GET /api/framework/shutdown/framework HTTP method to shut down myriad framework which shutdown web-server after stopping myriad driver, stop/clean all myriad tasks and clean myriad state-store (if any).
GET /api/framework/shutdown/framework
<resource_manager_host>:8192/api/framework/shutdown/framework
URL request example:
http://10.10.101.137:8192/api/framework/shutdown/framework
Curl request example:
curl http://10.10.101.137:8192/api/framework/shutdown/framework | python -m json.tool
Request header:
GET /api/framework/shutdown/framework HTTP/1.1 User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 Host: 10.10.101.137:8192 Accept: */*