blob: b776e21b0e4b355e5790ec856965fc228600d7ee [file] [log] [blame]
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Distributed File System-${project.version} - WebHDFS REST API
---
---
${maven.build.timestamp}
WebHDFS REST API
\[ {{{./index.html}Go Back}} \]
%{toc|section=1|fromDepth=0}
* {Document Conventions}
*----------------------+-------------------------------------------------------------------------------+
| <<<Monospaced>>> | Used for commands, HTTP request and responses and code blocks. |
*----------------------+-------------------------------------------------------------------------------+
| <<<\<Monospaced\>>>> | User entered values. |
*----------------------+-------------------------------------------------------------------------------+
| <<<[Monospaced]>>> | Optional values. When the value is not specified, the default value is used. |
*----------------------+-------------------------------------------------------------------------------+
| <Italics> | Important phrases and words. |
*----------------------+-------------------------------------------------------------------------------+
* {Introduction}
The HTTP REST API supports the complete
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}/{{{../../api/org/apache/hadoop/fs/FileContext.html}FileContext}}
interface for HDFS.
The operations and the corresponding FileSystem/FileContext methods are shown in the next section.
The Section {{HTTP Query Parameter Dictionary}} specifies the parameter details
such as the defaults and the valid values.
** {Operations}
* HTTP GET
* {{{Open and Read a File}<<<OPEN>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.open)
* {{{Status of a File/Directory}<<<GETFILESTATUS>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getFileStatus)
* {{<<<LISTSTATUS>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.listStatus)
* {{{Get Content Summary of a Directory}<<<GETCONTENTSUMMARY>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getContentSummary)
* {{{Get File Checksum}<<<GETFILECHECKSUM>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getFileChecksum)
* {{{Get Home Directory}<<<GETHOMEDIRECTORY>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getHomeDirectory)
* {{{Get Delegation Token}<<<GETDELEGATIONTOKEN>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getDelegationToken)
* {{{Get Delegation Tokens}<<<GETDELEGATIONTOKENS>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getDelegationTokens)
* HTTP PUT
* {{{Create and Write to a File}<<<CREATE>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.create)
* {{{Make a Directory}<<<MKDIRS>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.mkdirs)
* {{{Create a Symbolic Link}<<<CREATESYMLINK>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileContext.html}FileContext}}.createSymlink)
* {{{Rename a File/Directory}<<<RENAME>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.rename)
* {{{Set Replication Factor}<<<SETREPLICATION>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.setReplication)
* {{{Set Owner}<<<SETOWNER>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.setOwner)
* {{{Set Permission}<<<SETPERMISSION>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.setPermission)
* {{{Set Access or Modification Time}<<<SETTIMES>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.setTimes)
* {{{Renew Delegation Token}<<<RENEWDELEGATIONTOKEN>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.renewDelegationToken)
* {{{Cancel Delegation Token}<<<CANCELDELEGATIONTOKEN>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.cancelDelegationToken)
* HTTP POST
* {{{Append to a File}<<<APPEND>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.append)
* HTTP DELETE
* {{{Delete a File/Directory}<<<DELETE>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.delete)
** {FileSystem URIs vs HTTP URLs}
The FileSystem scheme of WebHDFS is "<<<webhdfs://>>>".
A WebHDFS FileSystem URI has the following format.
+---------------------------------
webhdfs://<HOST>:<HTTP_PORT>/<PATH>
+---------------------------------
The above WebHDFS URI corresponds to the below HDFS URI.
+---------------------------------
hdfs://<HOST>:<RPC_PORT>/<PATH>
+---------------------------------
In the REST API, the prefix "<<</webhdfs/v1>>>" is inserted in the path and a query is appended at the end.
Therefore, the corresponding HTTP URL has the following format.
+---------------------------------
http://<HOST>:<HTTP_PORT>/webhdfs/v1/<PATH>?op=...
+---------------------------------
** {HDFS Configuration Options}
Below are the HDFS configuration options for WebHDFS.
*-------------------------------------------------+---------------------------------------------------+
|| Property Name || Description |
*-------------------------------------------------+---------------------------------------------------+
| <<<dfs.webhdfs.enabled >>> | Enable/disable WebHDFS in Namenodes and Datanodes |
*-------------------------------------------------+---------------------------------------------------+
| <<<dfs.web.authentication.kerberos.principal>>> | The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint. The HTTP Kerberos principal MUST start with 'HTTP/' per Kerberos HTTP SPENGO specification. |
*-------------------------------------------------+---------------------------------------------------+
| <<<dfs.web.authentication.kerberos.keytab >>> | The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint. |
*-------------------------------------------------+---------------------------------------------------+
* {Authentication}
When security is <off>, the authenticated user is the username specified in the <<<user.name>>> query parameter.
If the <<<user.name>>> parameter is not set,
the server may either set the authenticated user to a default web user, if there is any, or return an error response.
When security is <on>, authentication is performed by either Hadoop delegation token or Kerberos SPNEGO.
If a token is set in the <<<delegation>>> query parameter, the authenticated user is the user encoded in the token.
If the <<<delegation>>> parameter is not set, the user is authenticated by Kerberos SPNEGO.
Below are examples using the <<<curl>>> command tool.
[[1]] Authentication when security is off:
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?[user.name=<USER>&]op=..."
+---------------------------------
[[1]] Authentication using Kerberos SPNEGO when security is on:
+---------------------------------
curl -i --negotiate -u : "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=..."
+---------------------------------
[[1]] Authentication using Hadoop delegation token when security is on:
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?delegation=<TOKEN>&op=..."
+---------------------------------
* {Proxy Users}
When the proxy user feature is enabled, a proxy user <P> may submit a request on behalf of another user <U>.
The username of <U> must be specified in the <<<doas>>> query parameter unless a delegation token is presented in authentication.
In such case, the information of both users <P> and <U> must be encoded in the delegation token.
[[1]] A proxy request when security is off:
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?[user.name=<USER>&]doas=<USER>&op=..."
+---------------------------------
[[1]] A proxy request using Kerberos SPNEGO when security is on:
+---------------------------------
curl -i --negotiate -u : "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?doas=<USER>&op=..."
+---------------------------------
[[1]] A proxy request using Hadoop delegation token when security is on:
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?delegation=<TOKEN>&op=..."
+---------------------------------
* {File and Directory Operations}
** {Create and Write to a File}
* Step 1: Submit a HTTP PUT request without automatically following redirects and without sending the file data.
+---------------------------------
curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE
[&overwrite=<true|false>][&blocksize=<LONG>][&replication=<SHORT>]
[&permission=<OCTAL>][&buffersize=<INT>]"
+---------------------------------
The request is redirected to a datanode where the file data is to be written:
+---------------------------------
HTTP/1.1 307 TEMPORARY_REDIRECT
Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE...
Content-Length: 0
+---------------------------------
* Step 2: Submit another HTTP PUT request using the URL in the <<<Location>>> header with the file data to be written.
+---------------------------------
curl -i -X PUT -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE..."
+---------------------------------
The client receives a <<<201 Created>>> response with zero content length
and the WebHDFS URI of the file in the <<<Location>>> header:
+---------------------------------
HTTP/1.1 201 Created
Location: webhdfs://<HOST>:<PORT>/<PATH>
Content-Length: 0
+---------------------------------
[]
<<Note>> that the reason of having two-step create/append is
for preventing clients to send out data before the redirect.
This issue is addressed by the "<<<Expect: 100-continue>>>" header in HTTP/1.1;
see {{{http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3}RFC 2616, Section 8.2.3}}.
Unfortunately, there are software library bugs (e.g. Jetty 6 HTTP server and Java 6 HTTP client),
which do not correctly implement "<<<Expect: 100-continue>>>".
The two-step create/append is a temporary workaround for the software library bugs.
See also:
{{{Overwrite}<<<overwrite>>>}},
{{{Block Size}<<<blocksize>>>}},
{{{Replication}<<<replication>>>}},
{{{Permission}<<<permission>>>}},
{{{Buffer Size}<<<buffersize>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.create
** {Append to a File}
* Step 1: Submit a HTTP POST request without automatically following redirects and without sending the file data.
+---------------------------------
curl -i -X POST "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=APPEND[&buffersize=<INT>]"
+---------------------------------
The request is redirected to a datanode where the file data is to be appended:
+---------------------------------
HTTP/1.1 307 TEMPORARY_REDIRECT
Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=APPEND...
Content-Length: 0
+---------------------------------
* Step 2: Submit another HTTP POST request using the URL in the <<<Location>>> header with the file data to be appended.
+---------------------------------
curl -i -X POST -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=APPEND..."
+---------------------------------
The client receives a response with zero content length:
+---------------------------------
HTTP/1.1 200 OK
Content-Length: 0
+---------------------------------
[]
See the note in the previous section for the description of why this operation requires two steps.
See also:
{{{Buffer Size}<<<buffersize>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.append
** {Open and Read a File}
* Submit a HTTP GET request with automatically following redirects.
+---------------------------------
curl -i -L "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN
[&offset=<LONG>][&length=<LONG>][&buffersize=<INT>]"
+---------------------------------
The request is redirected to a datanode where the file data can be read:
+---------------------------------
HTTP/1.1 307 TEMPORARY_REDIRECT
Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=OPEN...
Content-Length: 0
+---------------------------------
The client follows the redirect to the datanode and receives the file data:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 22
Hello, webhdfs user!
+---------------------------------
[]
See also:
{{{Offset}<<<offset>>>}},
{{{Length}<<<length>>>}},
{{{Buffer Size}<<<buffersize>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.open
** {Make a Directory}
* Submit a HTTP PUT request.
+---------------------------------
curl -i -X PUT "http://<HOST>:<PORT>/<PATH>?op=MKDIRS[&permission=<OCTAL>]"
+---------------------------------
The client receives a response with a {{{Boolean JSON Schema}<<<boolean>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"boolean": true}
+---------------------------------
[]
See also:
{{{Permission}<<<permission>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.mkdirs
** {Create a Symbolic Link}
* Submit a HTTP PUT request.
+---------------------------------
curl -i -X PUT "http://<HOST>:<PORT>/<PATH>?op=CREATESYMLINK
&destination=<PATH>[&createParent=<true|false>]"
+---------------------------------
The client receives a response with zero content length:
+---------------------------------
HTTP/1.1 200 OK
Content-Length: 0
+---------------------------------
[]
See also:
{{{Destination}<<<destination>>>}},
{{{Create Parent}<<<createParent>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.createSymlink
** {Rename a File/Directory}
* Submit a HTTP PUT request.
+---------------------------------
curl -i -X PUT "<HOST>:<PORT>/webhdfs/v1/<PATH>?op=RENAME&destination=<PATH>"
+---------------------------------
The client receives a response with a {{{Boolean JSON Schema}<<<boolean>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"boolean": true}
+---------------------------------
[]
See also:
{{{Destination}<<<destination>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.rename
** {Delete a File/Directory}
* Submit a HTTP DELETE request.
+---------------------------------
curl -i -X DELETE "http://<host>:<port>/webhdfs/v1/<path>?op=DELETE
[&recursive=<true|false>]"
+---------------------------------
The client receives a response with a {{{Boolean JSON Schema}<<<boolean>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"boolean": true}
+---------------------------------
[]
See also:
{{{Recursive}<<<recursive>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.delete
** {Status of a File/Directory}
* Submit a HTTP GET request.
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETFILESTATUS"
+---------------------------------
The client receives a response with a {{{FileStatus JSON Schema}<<<FileStatus>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{
"FileStatus":
{
"accessTime" : 0,
"blockSize" : 0,
"group" : "supergroup",
"length" : 0, //in bytes, zero for directories
"modificationTime": 1320173277227,
"owner" : "webuser",
"pathSuffix" : "",
"permission" : "777",
"replication" : 0,
"type" : "DIRECTORY" //enum {FILE, DIRECTORY, SYMLINK}
}
}
+---------------------------------
[]
See also:
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getFileStatus
** {List a Directory}
* Submit a HTTP GET request.
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=LISTSTATUS"
+---------------------------------
The client receives a response with a {{{FileStatuses JSON Schema}<<<FileStatuses>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 427
{
"FileStatuses":
{
"FileStatus":
[
{
"accessTime" : 1320171722771,
"blockSize" : 33554432,
"group" : "supergroup",
"length" : 24930,
"modificationTime": 1320171722771,
"owner" : "webuser",
"pathSuffix" : "a.patch",
"permission" : "644",
"replication" : 1,
"type" : "FILE"
},
{
"accessTime" : 0,
"blockSize" : 0,
"group" : "supergroup",
"length" : 0,
"modificationTime": 1320895981256,
"owner" : "szetszwo",
"pathSuffix" : "bar",
"permission" : "711",
"replication" : 0,
"type" : "DIRECTORY"
},
...
]
}
}
+---------------------------------
[]
See also:
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.listStatus
* {Other File System Operations}
** {Get Content Summary of a Directory}
* Submit a HTTP GET request.
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETCONTENTSUMMARY"
+---------------------------------
The client receives a response with a {{{ContentSummary JSON Schema}<<<ContentSummary>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{
"ContentSummary":
{
"directoryCount": 2,
"fileCount" : 1,
"length" : 24930,
"quota" : -1,
"spaceConsumed" : 24930,
"spaceQuota" : -1
}
}
+---------------------------------
[]
See also:
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getContentSummary
** {Get File Checksum}
* Submit a HTTP GET request.
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETFILECHECKSUM"
+---------------------------------
The request is redirected to a datanode:
+---------------------------------
HTTP/1.1 307 TEMPORARY_REDIRECT
Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=GETFILECHECKSUM...
Content-Length: 0
+---------------------------------
The client follows the redirect to the datanode and receives a {{{FileChecksum JSON Schema}<<<FileChecksum>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{
"FileChecksum":
{
"algorithm": "MD5-of-1MD5-of-512CRC32",
"bytes" : "eadb10de24aa315748930df6e185c0d ...",
"length" : 28
}
}
+---------------------------------
[]
See also:
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getFileChecksum
** {Get Home Directory}
* Submit a HTTP GET request.
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/?op=GETHOMEDIRECTORY"
+---------------------------------
The client receives a response with a {{{Path JSON Schema}<<<Path>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"Path": "/user/szetszwo"}
+---------------------------------
[]
See also:
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getHomeDirectory
** {Set Permission}
* Submit a HTTP PUT request.
+---------------------------------
curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETPERMISSION
[&permission=<OCTAL>]"
+---------------------------------
The client receives a response with zero content length:
+---------------------------------
HTTP/1.1 200 OK
Content-Length: 0
+---------------------------------
[]
See also:
{{{Permission}<<<permission>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.setPermission
** {Set Owner}
* Submit a HTTP PUT request.
+---------------------------------
curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETOWNER
[&owner=<USER>][&group=<GROUP>]"
+---------------------------------
The client receives a response with zero content length:
+---------------------------------
HTTP/1.1 200 OK
Content-Length: 0
+---------------------------------
[]
See also:
{{{Owner}<<<owner>>>}},
{{{Group}<<<group>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.setOwner
** {Set Replication Factor}
* Submit a HTTP PUT request.
+---------------------------------
curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETREPLICATION
[&replication=<SHORT>]"
+---------------------------------
The client receives a response with a {{{Boolean JSON Schema}<<<boolean>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"boolean": true}
+---------------------------------
[]
See also:
{{{Replication}<<<replication>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.setReplication
** {Set Access or Modification Time}
* Submit a HTTP PUT request.
+---------------------------------
curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETTIMES
[&modificationtime=<TIME>][&accesstime=<TIME>]"
+---------------------------------
The client receives a response with zero content length:
+---------------------------------
HTTP/1.1 200 OK
Content-Length: 0
+---------------------------------
[]
See also:
{{{Modification Time}<<<modificationtime>>>}},
{{{Access Time}<<<accesstime>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.setTimes
* {Delegation Token Operations}
** {Get Delegation Token}
* Submit a HTTP GET request.
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/?op=GETDELEGATIONTOKEN&renewer=<USER>"
+---------------------------------
The client receives a response with a {{{Token JSON Schema}<<<Token>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{
"Token":
{
"urlString": "JQAIaG9y..."
}
}
+---------------------------------
[]
See also:
{{{Renewer}<<<renewer>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getDelegationToken
** {Get Delegation Tokens}
* Submit a HTTP GET request.
+---------------------------------
curl -i "http://<HOST>:<PORT>/webhdfs/v1/?op=GETDELEGATIONTOKENS&renewer=<USER>"
+---------------------------------
The client receives a response with a {{{Tokens JSON Schema}<<<Tokens>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{
"Tokens":
{
"Token":
[
{
"urlString":"KAAKSm9i ..."
}
]
}
}
+---------------------------------
[]
See also:
{{{Renewer}<<<renewer>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getDelegationTokens
** {Renew Delegation Token}
* Submit a HTTP PUT request.
+---------------------------------
curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/?op=RENEWDELEGATIONTOKEN&token=<TOKEN>"
+---------------------------------
The client receives a response with a {{{Long JSON Schema}<<<long>>> JSON object}}:
+---------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"long": 1320962673997} //the new expiration time
+---------------------------------
[]
See also:
{{{Token}<<<token>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.renewDelegationToken
** {Cancel Delegation Token}
* Submit a HTTP PUT request.
+---------------------------------
curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/?op=CANCELDELEGATIONTOKEN&token=<TOKEN>"
+---------------------------------
The client receives a response with zero content length:
+---------------------------------
HTTP/1.1 200 OK
Content-Length: 0
+---------------------------------
[]
See also:
{{{Token}<<<token>>>}},
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.cancelDelegationToken
* {Error Responses}
When an operation fails, the server may throw an exception.
The JSON schema of error responses is defined in {{<<<RemoteException>>> JSON schema}}.
The table below shows the mapping from exceptions to HTTP response codes.
** {HTTP Response Codes}
*-------------------------------------+---------------------------------+
|| Exceptions || HTTP Response Codes |
*-------------------------------------+---------------------------------+
| <<<IllegalArgumentException >>> | <<<400 Bad Request >>> |
*-------------------------------------+---------------------------------+
| <<<UnsupportedOperationException>>> | <<<400 Bad Request >>> |
*-------------------------------------+---------------------------------+
| <<<SecurityException >>> | <<<401 Unauthorized >>> |
*-------------------------------------+---------------------------------+
| <<<IOException >>> | <<<403 Forbidden >>> |
*-------------------------------------+---------------------------------+
| <<<FileNotFoundException >>> | <<<404 Not Found >>> |
*-------------------------------------+---------------------------------+
| <<<RumtimeException >>> | <<<500 Internal Server Error>>> |
*-------------------------------------+---------------------------------+
Below are examples of exception responses.
*** {Illegal Argument Exception}
+---------------------------------
HTTP/1.1 400 Bad Request
Content-Type: application/json
Transfer-Encoding: chunked
{
"RemoteException":
{
"exception" : "IllegalArgumentException",
"javaClassName": "java.lang.IllegalArgumentException",
"message" : "Invalid value for webhdfs parameter \"permission\": ..."
}
}
+---------------------------------
*** {Security Exception}
+---------------------------------
HTTP/1.1 401 Unauthorized
Content-Type: application/json
Transfer-Encoding: chunked
{
"RemoteException":
{
"exception" : "SecurityException",
"javaClassName": "java.lang.SecurityException",
"message" : "Failed to obtain user group information: ..."
}
}
+---------------------------------
*** {Access Control Exception}
+---------------------------------
HTTP/1.1 403 Forbidden
Content-Type: application/json
Transfer-Encoding: chunked
{
"RemoteException":
{
"exception" : "AccessControlException",
"javaClassName": "org.apache.hadoop.security.AccessControlException",
"message" : "Permission denied: ..."
}
}
+---------------------------------
*** {File Not Found Exception}
+---------------------------------
HTTP/1.1 404 Not Found
Content-Type: application/json
Transfer-Encoding: chunked
{
"RemoteException":
{
"exception" : "FileNotFoundException",
"javaClassName": "java.io.FileNotFoundException",
"message" : "File does not exist: /foo/a.patch"
}
}
+---------------------------------
* {JSON Schemas}
All operations, except for {{{Open and Read a File}<<<OPEN>>>}},
either return a zero-length response or a JSON response.
For {{{Open and Read a File}<<<OPEN>>>}}, the response is an octet-stream.
The JSON schemas are shown below.
See {{{http://tools.ietf.org/id/draft-zyp-json-schema-03.html}draft-zyp-json-schema-03}}
for the syntax definitions of the JSON schemas.
** {Boolean JSON Schema}
+---------------------------------
{
"name" : "boolean",
"properties":
{
"boolean":
{
"description": "A boolean value",
"type" : "boolean",
"required" : true
}
}
}
+---------------------------------
See also:
{{{Make a Directory}<<<MKDIRS>>>}},
{{{Rename a File/Directory}<<<RENAME>>>}},
{{{Delete a File/Directory}<<<DELETE>>>}},
{{{Set Replication Factor}<<<SETREPLICATION>>>}}
** {ContentSummary JSON Schema}
+---------------------------------
{
"name" : "ContentSummary",
"properties":
{
"ContentSummary":
{
"type" : "object",
"properties":
{
"directoryCount":
{
"description": "The number of directories.",
"type" : "integer",
"required" : true
},
"fileCount":
{
"description": "The number of files.",
"type" : "integer",
"required" : true
},
"length":
{
"description": "The number of bytes used by the content.",
"type" : "integer",
"required" : true
},
"quota":
{
"description": "The namespace quota of this directory.",
"type" : "integer",
"required" : true
},
"spaceConsumed":
{
"description": "The disk space consumed by the content.",
"type" : "integer",
"required" : true
},
"spaceQuota":
{
"description": "The disk space quota.",
"type" : "integer",
"required" : true
}
}
}
}
}
+---------------------------------
See also:
{{{Get Content Summary of a Directory}<<<GETCONTENTSUMMARY>>>}}
** {FileChecksum JSON Schema}
+---------------------------------
{
"name" : "FileChecksum",
"properties":
{
"FileChecksum":
{
"type" : "object",
"properties":
{
"algorithm":
{
"description": "The name of the checksum algorithm.",
"type" : "string",
"required" : true
},
"bytes":
{
"description": "The byte sequence of the checksum in hexadecimal.",
"type" : "string",
"required" : true
},
"length":
{
"description": "The length of the bytes (not the length of the string).",
"type" : "integer",
"required" : true
}
}
}
}
}
+---------------------------------
See also:
{{{Get File Checksum}<<<GETFILECHECKSUM>>>}}
** {FileStatus JSON Schema}
+---------------------------------
{
"name" : "FileStatus",
"properties":
{
"FileStatus": fileStatusProperties //See FileStatus Properties
}
}
+---------------------------------
See also:
{{{FileStatus Properties}<<<FileStatus>>> Properties}},
{{{Status of a File/Directory}<<<GETFILESTATUS>>>}},
{{{../../api/org/apache/hadoop/fs/FileStatus}FileStatus}}
*** {FileStatus Properties}
JavaScript syntax is used to define <<<fileStatusProperties>>>
so that it can be referred in both <<<FileStatus>>> and <<<FileStatuses>>> JSON schemas.
+---------------------------------
var fileStatusProperties =
{
"type" : "object",
"properties":
{
"accessTime":
{
"description": "The access time.",
"type" : "integer",
"required" : true
},
"blockSize":
{
"description": "The block size of a file.",
"type" : "integer",
"required" : true
},
"group":
{
"description": "The group owner.",
"type" : "string",
"required" : true
},
"length":
{
"description": "The number of bytes in a file.",
"type" : "integer",
"required" : true
},
"modificationTime":
{
"description": "The modification time.",
"type" : "integer",
"required" : true
},
"owner":
{
"description": "The user who is the owner.",
"type" : "string",
"required" : true
},
"pathSuffix":
{
"description": "The path suffix.",
"type" : "string",
"required" : true
},
"permission":
{
"description": "The permission represented as a octal string.",
"type" : "string",
"required" : true
},
"replication":
{
"description": "The number of replication of a file.",
"type" : "integer",
"required" : true
},
"symlink": //an optional property
{
"description": "The link target of a symlink.",
"type" : "string"
},
"type":
{
"description": "The type of the path object.",
"enum" : ["FILE", "DIRECTORY", "SYMLINK"],
"required" : true
}
}
};
+---------------------------------
** {FileStatuses JSON Schema}
A <<<FileStatuses>>> JSON object represents an array of <<<FileStatus>>> JSON objects.
+---------------------------------
{
"name" : "FileStatuses",
"properties":
{
"FileStatuses":
{
"type" : "object",
"properties":
{
"FileStatus":
{
"description": "An array of FileStatus",
"type" : "array",
"items" : fileStatusProperties //See FileStatus Properties
}
}
}
}
}
+---------------------------------
See also:
{{{FileStatus Properties}<<<FileStatus>>> Properties}},
{{{List a Directory}<<<LISTSTATUS>>>}},
{{{../../api/org/apache/hadoop/fs/FileStatus}FileStatus}}
** {Long JSON Schema}
+---------------------------------
{
"name" : "long",
"properties":
{
"long":
{
"description": "A long integer value",
"type" : "integer",
"required" : true
}
}
}
+---------------------------------
See also:
{{{Renew Delegation Token}<<<RENEWDELEGATIONTOKEN>>>}},
** {Path JSON Schema}
+---------------------------------
{
"name" : "Path",
"properties":
{
"Path":
{
"description": "The string representation a Path.",
"type" : "string",
"required" : true
}
}
}
+---------------------------------
See also:
{{{Get Home Directory}<<<GETHOMEDIRECTORY>>>}},
{{{../../api/org/apache/hadoop/fs/Path}Path}}
** {RemoteException JSON Schema}
+---------------------------------
{
"name" : "RemoteException",
"properties":
{
"RemoteException":
{
"type" : "object",
"properties":
{
"exception":
{
"description": "Name of the exception",
"type" : "string",
"required" : true
},
"message":
{
"description": "Exception message",
"type" : "string",
"required" : true
},
"javaClassName": //an optional property
{
"description": "Java class name of the exception",
"type" : "string",
}
}
}
}
}
+---------------------------------
See also:
{{Error Responses}}
** {Token JSON Schema}
+---------------------------------
{
"name" : "Token",
"properties":
{
"Token": tokenProperties //See Token Properties
}
}
+---------------------------------
See also:
{{{Token Properties}<<<Token>>> Properties}},
{{{Get Delegation Token}<<<GETDELEGATIONTOKEN>>>}},
the note in {{Delegation}}.
*** {Token Properties}
JavaScript syntax is used to define <<<tokenProperties>>>
so that it can be referred in both <<<Token>>> and <<<Tokens>>> JSON schemas.
+---------------------------------
var tokenProperties =
{
"type" : "object",
"properties":
{
"urlString":
{
"description": "A delegation token encoded as a URL safe string.",
"type" : "string",
"required" : true
}
}
}
+---------------------------------
** {Tokens JSON Schema}
A <<<Tokens>>> JSON object represents an array of <<<Token>>> JSON objects.
+---------------------------------
{
"name" : "Tokens",
"properties":
{
"Tokens":
{
"type" : "object",
"properties":
{
"Token":
{
"description": "An array of Token",
"type" : "array",
"items" : "Token": tokenProperties //See Token Properties
}
}
}
}
}
+---------------------------------
See also:
{{{Token Properties}<<<Token>>> Properties}},
{{{Get Delegation Tokens}<<<GETDELEGATIONTOKENS>>>}},
the note in {{Delegation}}.
* {HTTP Query Parameter Dictionary}
** {Access Time}
*----------------+-------------------------------------------------------------------+
|| Name | <<<accesstime>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The access time of a file/directory. |
*----------------+-------------------------------------------------------------------+
|| Type | long |
*----------------+-------------------------------------------------------------------+
|| Default Value | -1 (means keeping it unchanged) |
*----------------+-------------------------------------------------------------------+
|| Valid Values | -1 or a timestamp |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any integer. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Set Access or Modification Time}<<<SETTIMES>>>}}
** {Block Size}
*----------------+-------------------------------------------------------------------+
|| Name | <<<blocksize>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The block size of a file. |
*----------------+-------------------------------------------------------------------+
|| Type | long |
*----------------+-------------------------------------------------------------------+
|| Default Value | Specified in the configuration. |
*----------------+-------------------------------------------------------------------+
|| Valid Values | \> 0 |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any integer. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Create and Write to a File}<<<CREATE>>>}}
** {Buffer Size}
*----------------+-------------------------------------------------------------------+
|| Name | <<<buffersize>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The size of the buffer used in transferring data. |
*----------------+-------------------------------------------------------------------+
|| Type | int |
*----------------+-------------------------------------------------------------------+
|| Default Value | Specified in the configuration. |
*----------------+-------------------------------------------------------------------+
|| Valid Values | \> 0 |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any integer. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Create and Write to a File}<<<CREATE>>>}},
{{{Append to a File}<<<APPEND>>>}},
{{{Open and Read a File}<<<OPEN>>>}}
** {Create Parent}
*----------------+-------------------------------------------------------------------+
|| Name | <<<createparent>>> |
*----------------+-------------------------------------------------------------------+
|| Description | If the parent directories do not exist, should they be created? |
*----------------+-------------------------------------------------------------------+
|| Type | boolean |
*----------------+-------------------------------------------------------------------+
|| Default Value | false |
*----------------+-------------------------------------------------------------------+
|| Valid Values | true | false |
*----------------+-------------------------------------------------------------------+
|| Syntax | true | false |
*----------------+-------------------------------------------------------------------+
See also:
{{{Create a Symbolic Link}<<<CREATESYMLINK>>>}}
** {Delegation}
*----------------+-------------------------------------------------------------------+
|| Name | <<<delegation>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The delegation token used for authentication. |
*----------------+-------------------------------------------------------------------+
|| Type | String |
*----------------+-------------------------------------------------------------------+
|| Default Value | \<empty\> |
*----------------+-------------------------------------------------------------------+
|| Valid Values | An encoded token. |
*----------------+-------------------------------------------------------------------+
|| Syntax | See the note below. |
*----------------+-------------------------------------------------------------------+
<<Note>> that delegation tokens are encoded as a URL safe string;
see <<<encodeToUrlString()>>>
and <<<decodeFromUrlString(String)>>>
in <<<org.apache.hadoop.security.token.Token>>> for the details of the encoding.
See also:
{{Authentication}}
** {Destination}
*----------------+-------------------------------------------------------------------+
|| Name | <<<destination>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The destination path. |
*----------------+-------------------------------------------------------------------+
|| Type | Path |
*----------------+-------------------------------------------------------------------+
|| Default Value | \<empty\> (an invalid path) |
*----------------+-------------------------------------------------------------------+
|| Valid Values | An absolute FileSystem path without scheme and authority. |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any path. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Create a Symbolic Link}<<<CREATESYMLINK>>>}},
{{{Rename a File/Directory}<<<RENAME>>>}}
** {Do As}
*----------------+-------------------------------------------------------------------+
|| Name | <<<doas>>> |
*----------------+-------------------------------------------------------------------+
|| Description | Allowing a proxy user to do as another user. |
*----------------+-------------------------------------------------------------------+
|| Type | String |
*----------------+-------------------------------------------------------------------+
|| Default Value | null |
*----------------+-------------------------------------------------------------------+
|| Valid Values | Any valid username. |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any string. |
*----------------+-------------------------------------------------------------------+
See also:
{{Proxy Users}}
** {Group}
*----------------+-------------------------------------------------------------------+
|| Name | <<<group>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The name of a group. |
*----------------+-------------------------------------------------------------------+
|| Type | String |
*----------------+-------------------------------------------------------------------+
|| Default Value | \<empty\> (means keeping it unchanged) |
*----------------+-------------------------------------------------------------------+
|| Valid Values | Any valid group name. |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any string. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Set Owner}<<<SETOWNER>>>}}
** {Length}
*----------------+-------------------------------------------------------------------+
|| Name | <<<length>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The number of bytes to be processed. |
*----------------+-------------------------------------------------------------------+
|| Type | long |
*----------------+-------------------------------------------------------------------+
|| Default Value | null (means the entire file) |
*----------------+-------------------------------------------------------------------+
|| Valid Values | \>= 0 or null |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any integer. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Open and Read a File}<<<OPEN>>>}}
** {Modification Time}
*----------------+-------------------------------------------------------------------+
|| Name | <<<modificationtime>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The modification time of a file/directory. |
*----------------+-------------------------------------------------------------------+
|| Type | long |
*----------------+-------------------------------------------------------------------+
|| Default Value | -1 (means keeping it unchanged) |
*----------------+-------------------------------------------------------------------+
|| Valid Values | -1 or a timestamp |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any integer. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Set Access or Modification Time}<<<SETTIMES>>>}}
** {Offset}
*----------------+-------------------------------------------------------------------+
|| Name | <<<offset>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The starting byte position. |
*----------------+-------------------------------------------------------------------+
|| Type | long |
*----------------+-------------------------------------------------------------------+
|| Default Value | 0 |
*----------------+-------------------------------------------------------------------+
|| Valid Values | \>= 0 |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any integer. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Open and Read a File}<<<OPEN>>>}}
** {Op}
*----------------+-------------------------------------------------------------------+
|| Name | <<<op>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The name of the operation to be executed. |
*----------------+-------------------------------------------------------------------+
|| Type | enum |
*----------------+-------------------------------------------------------------------+
|| Default Value | null (an invalid value) |
*----------------+-------------------------------------------------------------------+
|| Valid Values | Any valid operation name. |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any string. |
*----------------+-------------------------------------------------------------------+
See also:
{{Operations}}
** {Overwrite}
*----------------+-------------------------------------------------------------------+
|| Name | <<<overwrite>>> |
*----------------+-------------------------------------------------------------------+
|| Description | If a file already exists, should it be overwritten? |
*----------------+-------------------------------------------------------------------+
|| Type | boolean |
*----------------+-------------------------------------------------------------------+
|| Default Value | false |
*----------------+-------------------------------------------------------------------+
|| Valid Values | true | false |
*----------------+-------------------------------------------------------------------+
|| Syntax | true | false |
*----------------+-------------------------------------------------------------------+
See also:
{{{Create and Write to a File}<<<CREATE>>>}}
** {Owner}
*----------------+-------------------------------------------------------------------+
|| Name | <<<owner>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The username who is the owner of a file/directory. |
*----------------+-------------------------------------------------------------------+
|| Type | String |
*----------------+-------------------------------------------------------------------+
|| Default Value | \<empty\> (means keeping it unchanged) |
*----------------+-------------------------------------------------------------------+
|| Valid Values | Any valid username. |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any string. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Set Owner}<<<SETOWNER>>>}}
** {Permission}
*----------------+-------------------------------------------------------------------+
|| Name | <<<permission>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The permission of a file/directory. |
*----------------+-------------------------------------------------------------------+
|| Type | Octal |
*----------------+-------------------------------------------------------------------+
|| Default Value | 755 |
*----------------+-------------------------------------------------------------------+
|| Valid Values | 0 - 1777 |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any radix-8 integer (leading zeros may be omitted.) |
*----------------+-------------------------------------------------------------------+
See also:
{{{Create and Write to a File}<<<CREATE>>>}},
{{{Make a Directory}<<<MKDIRS>>>}},
{{{Set Permission}<<<SETPERMISSION>>>}}
** {Recursive}
*----------------+-------------------------------------------------------------------+
|| Name | <<<recursive>>> |
*----------------+-------------------------------------------------------------------+
|| Description | Should the operation act on the content in the subdirectories? |
*----------------+-------------------------------------------------------------------+
|| Type | boolean |
*----------------+-------------------------------------------------------------------+
|| Default Value | false |
*----------------+-------------------------------------------------------------------+
|| Valid Values | true | false |
*----------------+-------------------------------------------------------------------+
|| Syntax | true | false |
*----------------+-------------------------------------------------------------------+
See also:
{{{Rename a File/Directory}<<<RENAME>>>}}
** {Renewer}
*----------------+-------------------------------------------------------------------+
|| Name | <<<renewer>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The username of the renewer of a delegation token. |
*----------------+-------------------------------------------------------------------+
|| Type | String |
*----------------+-------------------------------------------------------------------+
|| Default Value | \<empty\> (means the current user) |
*----------------+-------------------------------------------------------------------+
|| Valid Values | Any valid username. |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any string. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Get Delegation Token}<<<GETDELEGATIONTOKEN>>>}},
{{{Get Delegation Tokens}<<<GETDELEGATIONTOKENS>>>}}
** {Replication}
*----------------+-------------------------------------------------------------------+
|| Name | <<<replication>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The number of replications of a file. |
*----------------+-------------------------------------------------------------------+
|| Type | short |
*----------------+-------------------------------------------------------------------+
|| Default Value | Specified in the configuration. |
*----------------+-------------------------------------------------------------------+
|| Valid Values | \> 0 |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any integer. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Create and Write to a File}<<<CREATE>>>}},
{{{Set Replication Factor}<<<SETREPLICATION>>>}}
** {Token}
*----------------+-------------------------------------------------------------------+
|| Name | <<<token>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The delegation token used for the operation. |
*----------------+-------------------------------------------------------------------+
|| Type | String |
*----------------+-------------------------------------------------------------------+
|| Default Value | \<empty\> |
*----------------+-------------------------------------------------------------------+
|| Valid Values | An encoded token. |
*----------------+-------------------------------------------------------------------+
|| Syntax | See the note in {{Delegation}}. |
*----------------+-------------------------------------------------------------------+
See also:
{{{Renew Delegation Token}<<<RENEWDELEGATIONTOKEN>>>}},
{{{Cancel Delegation Token}<<<CANCELDELEGATIONTOKEN>>>}}
** {Username}
*----------------+-------------------------------------------------------------------+
|| Name | <<<user.name>>> |
*----------------+-------------------------------------------------------------------+
|| Description | The authenticated user; see {{Authentication}}. |
*----------------+-------------------------------------------------------------------+
|| Type | String |
*----------------+-------------------------------------------------------------------+
|| Default Value | null |
*----------------+-------------------------------------------------------------------+
|| Valid Values | Any valid username. |
*----------------+-------------------------------------------------------------------+
|| Syntax | Any string. |
*----------------+-------------------------------------------------------------------+
See also:
{{Authentication}}