title: “Specification” linkTitle: “Specification” weight: 4 date: 2021-10-25 aliases:
This document defines Apache Avro. It is intended to be the authoritative specification. Implementations of Avro must adhere to this document.
A Schema is represented in JSON by one of:
{"type": "typeName", ...attributes...}
where typeName is either a primitive or derived type name, as defined below. Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.
The set of primitive type names is:
Primitive types have no specified attributes.
Primitive type names are also defined type names. Thus, for example, the schema “string” is equivalent to:
{"type": "string"}
Avro supports six kinds of complex types: records, enums, arrays, maps, unions and fixed.
Records use the type name “record” and support the following attributes:
field default values
avro type | json type | example |
---|---|---|
null | null | null |
boolean | boolean | true |
int,long | integer | 1 |
float,double | number | 1.1 |
bytes | string | "\u00FF" |
string | string | "foo" |
record | object | {"a": 1} |
enum | string | "FOO" |
array | array | [1] |
map | object | {"a": 1} |
fixed | string | "\u00ff" |
For example, a linked-list of 64-bit values may be defined with:
{ "type": "record", "name": "LongList", "aliases": ["LinkedLongs"], // old name for this "fields" : [ {"name": "value", "type": "long"}, // each element has a long {"name": "next", "type": ["null", "LongList"]} // optional next element ] }
Enums use the type name “enum” and support the following attributes:
For example, playing card suits might be defined with:
{ "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS", "DIAMONDS", "CLUBS"] }
Arrays use the type name “array” and support a single attribute:
For example, an array of strings is declared with:
{ "type": "array", "items" : "string", "default": [] }
Maps use the type name “map” and support one attribute:
Map keys are assumed to be strings.
For example, a map from string to long is declared with:
{ "type": "map", "values" : "long", "default": {} }
Unions, as mentioned above, are represented using JSON arrays. For example, ["null", "string"]
declares a schema which may be either a null or string.
(Note that when a [default value]({{< ref “#schema-record” >}} “Schema record”) is specified for a record field whose type is a union, the type of the default value must match with one element of the union.
Unions may not contain more than one schema with the same type, except for the named types record, fixed and enum. For example, unions containing two array types or two map types are not permitted, but two types with different names are permitted. (Names permit efficient resolution when reading and writing unions.)
Unions may not immediately contain other unions.
Fixed uses the type name “fixed” and supports the following attributes:
For example, 16-byte quantity may be declared with:
{"type": "fixed", "size": 16, "name": "md5"}
Record, enums and fixed are named types. Each has a fullname that is composed of two parts: a name and a namespace, separated by a dot. Equality of names is defined on the fullname – it is an error to specify two different types with the same name.
Record fields and enum symbols have names as well (but no namespace). Equality of field names and enum symbols is defined within their scope (the record/enum that defines them). It is an error to define multiple fields or enum symbols with the same name in a single type. Fields and enum symbols across scopes are never equal, so field names and enum symbols can be reused in a different type.
The name portion of the fullname of named types, record field names, and enum symbols must:
A namespace is a dot-separated sequence of such names. The empty string may also be used as a namespace to indicate the null namespace. Equality of names (including field names and enum symbols) as well as fullnames is case-sensitive.
The null namespace may not be used in a dot-separated sequence of names. So the grammar for a namespace is:
<empty> | <name>[(<dot><name>)*]
In record, enum and fixed definitions, the fullname is determined according to the algorithm below the example:
{ "type": "record", "name": "Example", "doc": "A simple name (attribute) and no namespace attribute: use the null namespace (\"\"); the fullname is 'Example'.", "fields": [ { "name": "inheritNull", "type": { "type": "enum", "name": "Simple", "doc": "A simple name (attribute) and no namespace attribute: inherit the null namespace of the enclosing type 'Example'. The fullname is 'Simple'.", "symbols": ["a", "b"] } }, { "name": "explicitNamespace", "type": { "type": "fixed", "name": "Simple", "namespace": "explicit", "doc": "A simple name (attribute) and a namespace (attribute); the fullname is 'explicit.Simple' (this is a different type than of the 'inheritNull' field).", "size": 12 } }, { "name": "fullName", "type": { "type": "record", "name": "a.full.Name", "namespace": "ignored", "doc": "A name attribute with a fullname, so the namespace attribute is ignored. The fullname is 'a.full.Name', and the namespace is 'a.full'.", "fields": [ { "name": "inheritNamespace", "type": { "type": "enum", "name": "Understanding", "doc": "A simple name (attribute) and no namespace attribute: inherit the namespace of the enclosing type 'a.full.Name'. The fullname is 'a.full.Understanding'.", "symbols": ["d", "e"] } } ] } } ] }
The fullname of a record, enum or fixed definition is determined by the required name
and optional namespace
attributes like this:
References to previously defined names are as in the latter two cases above: if they contain a dot they are a fullname, if they do not contain a dot, the namespace is the namespace of the enclosing definition.
Primitive type names (null
, boolean
, int
, long
, float
, double
, bytes
, string
) have no namespace and their names may not be defined in any namespace.
Complex types (record
, enum
, array
, map
, fixed
) have no namespace, but their names (as well as union
) are permitted to be reused as type names. This can be confusing to the human reader, but is always unambiguous for binary serialization. Due to the limitations of JSON encoding, it is a best practice to use a namespace when using these names.
A schema or protocol may not contain multiple definitions of a fullname. Further, a name must be defined before it is used (“before” in the depth-first, left-to-right traversal of the JSON parse tree, where the types attribute of a protocol is always deemed to come “before” the messages attribute.)
Named types and fields may have aliases. An implementation may optionally use aliases to map a writer‘s schema to the reader’s. This facilitates both schema evolution as well as processing disparate datasets.
Aliases function by re-writing the writer‘s schema using aliases from the reader’s schema. For example, if the writer‘s schema was named “Foo” and the reader’s schema is named “Bar” and has an alias of “Foo”, then the implementation would act as though “Foo” were named “Bar” when reading. Similarly, if data was written as a record with a field named “x” and is read as a record with a field named “y” with alias “x”, then the implementation would act as though “x” were named “y” when reading.
A type alias may be specified either as a fully namespace-qualified, or relative to the namespace of the name it is an alias for. For example, if a type named “a.b” has aliases of “c” and “x.y”, then the fully qualified names of its aliases are “a.c” and “x.y”.
Aliases are alternative names, and thus subject to the same uniqueness constraints as names. Aliases should be valid names, but this is not required: any string is accepted as an alias. When aliases are used “to map a writer‘s schema to the reader’s” (see above), this allows schema evolution to correct illegal names in old schemata.
Over time, rules and validations on schemas have changed. It is therefore possible that a schema used to work with an older version of Avro, but now fails to parse.
This can have several reasons, as listed below. Each reason also describes a fix, which can be applied using [schema resolution]({{< ref “#schema-resolution” >}}): you fix the problems in the schema in a way that is compatible, and then you can use the new schema to read the old data.
Invalid names of types and fields can be corrected by renaming (using an [alias]({{< ref “#aliases” >}})). This works for simple names, namespaces and fullnames.
This fix is twofold: first, you add the invalid name as an alias to the type/field. Then, you change the name to any valid name.
Default values are only used to fill in missing data when reading. Invalid defaults create invalid values in these cases. The fix is to correct the default values.
Binary encoded Avro data does not include type information or field names. The benefit is that the serialized data is small, but as a result a schema must always be used in order to read Avro data correctly. The best way to ensure that the schema is structurally identical to the one used to write the data is to use the exact same schema.
Therefore, files or systems that store Avro data should always include the writer's schema for that data. Avro-based remote procedure call (RPC) systems must also guarantee that remote recipients of data have a copy of the schema used to write that data. In general, it is advisable that any reader of Avro data should use a schema that is the same (as defined more fully in [Parsing Canonical Form for Schemas]({{< ref “#parsing-canonical-form-for-schemas” >}} “Parsing Canonical Form for Schemas”)) as the schema that was used to write the data in order to deserialize it correctly. Deserializing data into a newer schema is accomplished by specifying an additional schema, the results of which are described in [Schema Resolution]({{< ref “#schema-resolution” >}}).
In general, both serialization and deserialization proceed as a depth-first, left-to-right traversal of the schema, serializing or deserializing primitive types as they are encountered. Therefore, it is possible, though not advisable, to read Avro data with a schema that does not have the same Parsing Canonical Form as the schema with which the data was written. In order for this to work, the serialized primitive values must be compatible, in order value by value, with the items in the deserialization schema. For example, int and long are always serialized the same way, so an int could be deserialized as a long. Since the compatibility of two schemas depends on both the data and the serialization format (eg. binary is more permissive than JSON because JSON includes field names, eg. a long that is too large will overflow an int), it is simpler and more reliable to use schemas with identical Parsing Canonical Form.
Avro specifies two serialization encodings: binary and JSON. Most applications will use the binary encoding, as it is smaller and faster. But, for debugging and web-based applications, the JSON encoding may sometimes be appropriate.
Binary encoding does not include field names, self-contained information about the types of individual bytes, nor field or record separators. Therefore readers are wholly reliant on the schema used when the data was encoded.
Primitive types are encoded in binary as follows:
value | hex |
---|---|
0 | 00 |
-1 | 01 |
1 | 02 |
-2 | 03 |
2 | 04 |
... | ... |
-64 | 7f |
64 | 80 01 |
... | ... |
06 66 6f 6f
Complex types are encoded in binary as follows:
A record is encoded by encoding the values of its fields in the order that they are declared. In other words, a record is encoded as just the concatenation of the encodings of its fields. Field values are encoded per their schema.
For example, the record schema
{ "type": "record", "name": "test", "fields" : [ {"name": "a", "type": "long"}, {"name": "b", "type": "string"} ] }
An instance of this record whose a field has value 27 (encoded as hex 36) and whose b field has value “foo” (encoded as hex bytes 06 66 6f 6f), would be encoded simply as the concatenation of these, namely the hex byte sequence:
36 06 66 6f 6f
An enum is encoded by a int, representing the zero-based position of the symbol in the schema.
For example, consider the enum:
{"type": "enum", "name": "Foo", "symbols": ["A", "B", "C", "D"] }
This would be encoded by an int between zero and three, with zero indicating “A”, and 3 indicating “D”.
Arrays are encoded as a series of blocks. Each block consists of a long count value, followed by that many array items. A block with count zero indicates the end of the array. Each item is encoded per the array's item schema.
If a block's count is negative, its absolute value is used, and the count is followed immediately by a long block size indicating the number of bytes in the block. This block size permits fast skipping through data, e.g., when projecting a record to a subset of its fields.
For example, the array schema
{"type": "array", "items": "long"}
an array containing the items 3 and 27 could be encoded as the long value 2 (encoded as hex 04) followed by long values 3 and 27 (encoded as hex 06 36) terminated by zero:
04 06 36 00
The blocked representation permits one to read and write arrays larger than can be buffered in memory, since one can start writing items without knowing the full length of the array.
Maps are encoded as a series of blocks. Each block consists of a long
count value, followed by that many key/value pairs. A block with count zero indicates the end of the map. Each item is encoded per the map's value schema.
If a block's count is negative, its absolute value is used, and the count is followed immediately by a long
block size indicating the number of bytes in the block. This block size permits fast skipping through data, e.g., when projecting a record to a subset of its fields.
The blocked representation permits one to read and write maps larger than can be buffered in memory, since one can start writing items without knowing the full length of the map.
A union is encoded by first writing an int
value indicating the zero-based position within the union of the schema of its value. The value is then encoded per the indicated schema within the union.
For example, the union schema ["null","string"]
would encode:
00
02 02 61
NOTE: Currently for C/C++ implementations, the positions are practically an int, but theoretically a long. In reality, we don't expect unions with 215M membersFixed instances are encoded using the number of bytes declared in the schema.
Except for unions, the JSON encoding is the same as is used to encode [field default values]({{< ref “#schema-record” >}}).
The value of a union is encoded in JSON as follows:
For example, the union schema ["null","string","Foo"]
, where Foo is a record name, would encode:
{"string": "a"}
and{"Foo": {...}}
, where {...}
indicates the JSON encoding of a Foo instance.Note that the original schema is still required to correctly process JSON-encoded data. For example, the JSON encoding does not distinguish between int and long, float and double, records and maps, enums and strings, etc.
In some situations a single Avro serialized object is to be stored for a longer period of time. One very common example is storing Avro records for several weeks in an Apache Kafka topic.
In the period after a schema change this persistence system will contain records that have been written with different schemas. So the need arises to know which schema was used to write a record to support schema evolution correctly. In most cases the schema itself is too large to include in the message, so this binary wrapper format supports the use case more effectively.
Single Avro objects are encoded as follows:
C3 01
, to show that the message is Avro and uses this single-record format (version 1).Implementations use the 2-byte marker to determine whether a payload is Avro. This check helps avoid expensive lookups that resolve the schema from a fingerprint, when the message is not an encoded Avro payload.
Avro defines a standard sort order for data. This permits data written by one system to be efficiently sorted by another system. This can be an important optimization, as sort order comparisons are sometimes the most frequent per-object operation. Note also that Avro binary-encoded data can be efficiently ordered without deserializing it to objects.
Data items may only be compared if they have identical schemas. Pairwise comparisons are implemented recursively with a depth-first, left-to-right traversal of the schema. The first mismatch encountered determines the order of the items.
Two items with the same schema are compared according to the following rules.
["z", "a"]
would sort “z” values before “a” values.["int", "string"]
union would order all int values before all string values, with the ints and strings themselves ordered as defined above."order":"ignore"
record field.Avro includes a simple object container file format. A file has a schema, and all objects stored in the file must be written according to that schema, using binary encoding. Objects are stored in blocks that may be compressed. Syncronization markers are used between blocks to permit efficient splitting of files for MapReduce processing.
Files may include arbitrary user-specified metadata.
A file consists of:
A file header consists of:
File metadata is written as if defined by the following [map]({{< ref “#schema-maps” >}}) schema:
{"type": "map", "values": "bytes"}
All metadata properties that start with “avro.” are reserved. The following file metadata properties are currently used:
A file header is thus described by the following schema:
{"type": "record", "name": "org.apache.avro.file.Header", "fields" : [ {"name": "magic", "type": {"type": "fixed", "name": "Magic", "size": 4}}, {"name": "meta", "type": {"type": "map", "values": "bytes"}}, {"name": "sync", "type": {"type": "fixed", "name": "Sync", "size": 16}} ] }
A file data block consists of:
A file data block is thus described by the following schema:
{"type": "record", "name": "org.apache.avro.file.DataBlock", "fields" : [ {"name": "count", "type": "long"}, {"name": "data", "type": "bytes"}, {"name": "sync", "type": {"type": "fixed", "name": "Sync", "size": 16}} ] }
Each block's binary data can be efficiently extracted or skipped without deserializing the contents. The combination of block size, object counts, and sync markers enable detection of corrupt blocks and help ensure data integrity.
null
The “null” codec simply passes through data uncompressed.
deflate
The “deflate” codec writes the data block using the deflate algorithm as specified in RFC 1951, and typically implemented using the zlib library. Note that this format (unlike the “zlib format” in RFC 1950) does not have a checksum.
bzip2
The “bzip2” codec uses the bzip2 compression library.
snappy
The “snappy” codec uses Google's Snappy compression library. Each compressed block is followed by the 4-byte, big-endian CRC32 checksum of the uncompressed data in the block.
xz
The “xz” codec uses the XZ compression library.
zstandard
The “zstandard” codec uses Facebook's Zstandard compression library.
Avro protocols describe RPC interfaces. Like schemas, they are defined with JSON text.
A protocol is a JSON object with the following attributes:
The name and namespace qualification rules defined for schema objects apply to protocols as well.
A message has attributes:
["AccessError"]
, then the effective union is ["string", "AccessError"]
. When no errors are declared, the effective error union is ["string"]
. Errors are serialized using the effective union; however, a protocol's JSON declaration contains only the declared union.A request parameter list is processed equivalently to an anonymous record. Since record field lists may vary between reader and writer, request parameters may also differ between the caller and responder, and such differences are resolved in the same manner as record field differences.
The one-way parameter may only be true when the response type is "null"
and no errors are listed.
For example, one may define a simple HelloWorld protocol with:
{ "namespace": "com.acme", "protocol": "HelloWorld", "doc": "Protocol Greetings", "types": [ {"name": "Greeting", "type": "record", "fields": [ {"name": "message", "type": "string"}]}, {"name": "Curse", "type": "error", "fields": [ {"name": "message", "type": "string"}]} ], "messages": { "hello": { "doc": "Say hello.", "request": [{"name": "greeting", "type": "Greeting" }], "response": "Greeting", "errors": ["Curse"] } } }
Messages may be transmitted via different transport mechanisms.
To the transport, a message is an opaque byte sequence.
A transport is a system that supports:
Transports may be either stateless or stateful. In a stateless transport, messaging assumes no established connection state, while stateful transports establish connections that may be used for multiple messages. This distinction is discussed further in the handshake section below.
When HTTP is used as a transport, each Avro message exchange is an HTTP request/response pair. All messages of an Avro protocol should share a single URL at an HTTP server. Other protocols may also use that URL. Both normal and error Avro response messages should use the 200 (OK) response code. The chunked encoding may be used for requests and responses, but, regardless the Avro request and response are the entire content of an HTTP request and response. The HTTP Content-Type of requests and responses should be specified as “avro/binary”. Requests should be made using the POST method.
HTTP is used by Avro as a stateless transport.
Avro messages are framed as a list of buffers.
Framing is a layer between messages and the transport. It exists to optimize certain operations.
The format of framed message data is:
Framing is transparent to request and response message formats (described below). Any message may be presented as a single or multiple buffers.
Framing can permit readers to more efficiently get different buffers from different sources and for writers to more efficiently store different buffers to different destinations. In particular, it can reduce the number of times large binary objects are copied. For example, if an RPC parameter consists of a megabyte of file data, that data can be copied directly to a socket from a file descriptor, and, on the other end, it could be written directly to a file descriptor, never entering user space.
A simple, recommended, framing policy is for writers to create a new segment whenever a single binary object is written that is larger than a normal output buffer. Small objects are then appended in buffers, while larger objects are written as their own buffers. When a reader then tries to read a large object the runtime can hand it an entire buffer directly, without having to copy it.
The purpose of the handshake is to ensure that the client and the server have each other's protocol definition, so that the client can correctly deserialize responses, and the server can correctly deserialize requests. Both clients and servers should maintain a cache of recently seen protocols, so that, in most cases, a handshake will be completed without extra round-trip network exchanges or the transmission of full protocol text.
RPC requests and responses may not be processed until a handshake has been completed. With a stateless transport, all requests and responses are prefixed by handshakes. With a stateful transport, handshakes are only attached to requests and responses until a successful handshake response has been returned over a connection. After this, request and response payloads are sent without handshakes for the lifetime of that connection.
The handshake process uses the following record schemas:
{ "type": "record", "name": "HandshakeRequest", "namespace":"org.apache.avro.ipc", "fields": [ {"name": "clientHash", "type": {"type": "fixed", "name": "MD5", "size": 16}}, {"name": "clientProtocol", "type": ["null", "string"]}, {"name": "serverHash", "type": "MD5"}, {"name": "meta", "type": ["null", {"type": "map", "values": "bytes"}]} ] } { "type": "record", "name": "HandshakeResponse", "namespace": "org.apache.avro.ipc", "fields": [ {"name": "match", "type": {"type": "enum", "name": "HandshakeMatch", "symbols": ["BOTH", "CLIENT", "NONE"]}}, {"name": "serverProtocol", "type": ["null", "string"]}, {"name": "serverHash", "type": ["null", {"type": "fixed", "name": "MD5", "size": 16}]}, {"name": "meta", "type": ["null", {"type": "map", "values": "bytes"}]} ] }
HandshakeRequest
containing just the hash of its protocol and of the server‘s protocol (clientHash!=null, clientProtocol=null, serverHash!=null
), where the hashes are 128-bit MD5 hashes of the JSON protocol text. If a client has never connected to a given server, it sends its hash as a guess of the server’s hash, otherwise it sends the hash that it previously obtained from this server. The server responds with a HandshakeResponse containing one of:match=BOTH, serverProtocol=null, serverHash=null
if the client sent the valid hash of the server‘s protocol and the server knows what protocol corresponds to the client’s hash. In this case, the request is complete and the response data immediately follows the HandshakeResponse.match=CLIENT, serverProtocol!=null, serverHash!=null
if the server has previously seen the client‘s protocol, but the client sent an incorrect hash of the server’s protocol. The request is complete and the response data immediately follows the HandshakeResponse. The client must use the returned protocol to process the response and should also cache that protocol and its hash for future interactions with this server.match=NONE
if the server has not previously seen the client‘s protocol. The serverHash and serverProtocol may also be non-null if the server’s protocol hash was incorrect. In this case the client must then re-submit its request with its protocol text (clientHash!=null, clientProtocol!=null, serverHash!=null
) and the server should respond with a successful match (match=BOTH, serverProtocol=null, serverHash=null) as above.The meta field is reserved for future handshake enhancements.
A call consists of a request message paired with its resulting response or error message. Requests and responses contain extensible metadata, and both kinds of messages are framed as described above.
The format of a call request is:
When a message is declared one-way and a stateful connection has been established by a successful handshake response, no response data is sent. Otherwise the format of the call response is:
A reader of Avro data, whether from an RPC or a file, can always parse that data because the original schema must be provided along with the data. However, the reader may be programmed to read data into a different schema. For example, if the data was written with a different version of the software than it is read, then fields may have been added or removed from records. This section specifies how such schema differences should be resolved.
We refer to the schema used to write the data as the writer‘s schema, and the schema that the application expects the reader’s schema. Differences between these should be resolved as follows:
It is an error if the two schemas do not match. To match, one of the following must hold:
if both are records:
if both are enums: if the writer‘s symbol is not present in the reader’s enum and the reader has a default value, then that value is used, otherwise an error is signalled.
if both are arrays: This resolution algorithm is applied recursively to the reader‘s and writer’s array item schemas.
if both are maps: This resolution algorithm is applied recursively to the reader‘s and writer’s value schemas.
if both are unions: The first schema in the reader‘s union that matches the selected writer’s union schema is recursively resolved against it. if none match, an error is signalled.
if reader‘s is a union, but writer’s is not The first schema in the reader‘s union that matches the writer’s schema is recursively resolved against it. If none match, an error is signalled.
if writer‘s is a union, but reader’s is not If the reader‘s schema matches the selected writer’s schema, it is recursively resolved against it. If they do not match, an error is signalled.
A schema's doc fields are ignored for the purposes of schema resolution. Hence, the doc portion of a schema may be dropped at serialization.
One of the defining characteristics of Avro is that a reader must use the schema used by the writer of the data in order to know how to read the data. This assumption results in a data format that's compact and also amenable to many forms of schema evolution. However, the specification so far has not defined what it means for the reader to have the “same” schema as the writer. Does the schema need to be textually identical? Well, clearly adding or removing some whitespace to a JSON expression does not change its meaning. At the same time, reordering the fields of records clearly does change the meaning. So what does it mean for a reader to have “the same” schema as a writer?
Parsing Canonical Form is a transformation of a writer‘s schema that let’s us define what it means for two schemas to be “the same” for the purpose of reading data written against the schema. It is called Parsing Canonical Form because the transformations strip away parts of the schema, like “doc” attributes, that are irrelevant to readers trying to parse incoming data. It is called Canonical Form because the transformations normalize the JSON text (such as the order of attributes) in a way that eliminates unimportant differences between schemas. If the Parsing Canonical Forms of two different schemas are textually equal, then those schemas are “the same” as far as any reader is concerned, i.e., there is no serialized data that would allow a reader to distinguish data generated by a writer using one of the original schemas from data generated by a writing using the other original schema. (We sketch a proof of this property in a companion document.)
The next subsection specifies the transformations that define Parsing Canonical Form. But with a well-defined canonical form, it can be convenient to go one step further, transforming these canonical forms into simple integers (“fingerprints”) that can be used to uniquely identify schemas. The subsection after next recommends some standard practices for generating such fingerprints.
Assuming an input schema (in JSON form) that's already UTF-8 text for a valid Avro schema (including all quotes as required by JSON), the following transformations will produce its Parsing Canonical Form:
{"type":"int"}
).“[A] fingerprinting algorithm is a procedure that maps an arbitrarily large data item (such as a computer file) to a much shorter bit string, its fingerprint, that uniquely identifies the original data for all practical purposes” (quoted from Wikipedia). In the Avro context, fingerprints of Parsing Canonical Form can be useful in a number of applications; for example, to cache encoder and decoder objects, to tag data items with a short substitute for the writer's full schema, and to quickly negotiate common-case schemas between readers and writers.
In designing fingerprinting algorithms, there is a fundamental trade-off between the length of the fingerprint and the probability of collisions. To help application designers find appropriate points within this trade-off space, while encouraging interoperability and ease of implementation, we recommend using one of the following three algorithms when fingerprinting Avro schemas:
These fingerprints are not meant to provide any security guarantees, even the longer SHA-256-based ones. Most Avro applications should be surrounded by security measures that prevent attackers from writing random data and otherwise interfering with the consumers of schemas. We recommend that these surrounding mechanisms be used to prevent collision and pre-image attacks (i.e., “forgery”) on schema fingerprints, rather than relying on the security properties of the fingerprints themselves.
Rabin fingerprints are cyclic redundancy checks computed using irreducible polynomials. In the style of the Appendix of RFC 1952 (pg 10), which defines the CRC-32 algorithm, here's our definition of the 64-bit AVRO fingerprinting algorithm:
long fingerprint64(byte[] buf) { if (FP_TABLE == null) initFPTable(); long fp = EMPTY; for (int i = 0; i < buf.length; i++) fp = (fp >>> 8) ^ FP_TABLE[(int)(fp ^ buf[i]) & 0xff]; return fp; } static long EMPTY = 0xc15d213aa4d7a795L; static long[] FP_TABLE = null; void initFPTable() { FP_TABLE = new long[256]; for (int i = 0; i < 256; i++) { long fp = i; for (int j = 0; j < 8; j++) fp = (fp >>> 1) ^ (EMPTY & -(fp & 1L)); FP_TABLE[i] = fp; } }
Readers interested in the mathematics behind this algorithm may want to read Chapter 14 of the Second Edition of Hacker's Delight. (Unlike RFC-1952 and the book chapter, we prepend a single one bit to messages. We do this because CRCs ignore leading zero bits, which can be problematic. Our code prepends a one-bit by initializing fingerprints using EMPTY, rather than initializing using zero as in RFC-1952 and the book chapter.)
A logical type is an Avro primitive or complex type with extra attributes to represent a derived type. The attribute logicalType
must always be present for a logical type, and is a string with the name of one of the logical types listed later in this section. Other attributes may be defined for particular logical types.
A logical type is always serialized using its underlying Avro type so that values are encoded in exactly the same way as the equivalent Avro type that does not have a logicalType
attribute. Language implementations may choose to represent logical types with an appropriate native type, although this is not required.
Language implementations must ignore unknown logical types when reading, and should use the underlying Avro type. If a logical type is invalid, for example a decimal with scale greater than its precision, then implementations should ignore the logical type and use the underlying Avro type.
The decimal
logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
A decimal
logical type annotates Avro bytes or fixed types. The byte array must contain the two's-complement representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified using an attribute.
The following attributes are supported:
{ "type": "bytes", "logicalType": "decimal", "precision": 4, "scale": 2 }
Precision must be a positive integer greater than zero. If the underlying type is a fixed, then the precision is limited by its size. An array of length n can store at most floor(log10(28 × n - 1 - 1)) base-10 digits of precision.
Scale must be zero or a positive integer less than or equal to the precision.
For the purposes of schema resolution, two schemas that are decimal
logical types match if their scales and precisions match.
alternative
As it's not always possible to fix scale and precision in advance for a decimal field, big-decimal
is another decimal
logical type restrict to Avro bytes.
Currently only available in Java and Rust.
{ "type": "bytes", "logicalType": "big-decimal" }
Here, as scale property is stored in value itself it needs more bytes than preceding decimal
type, but it allows more flexibility.
The uuid
logical type represents a random generated universally unique identifier (UUID).
A uuid
logical type annotates an Avro string
. The string has to conform with RFC-4122
The date
logical type represents a date within the calendar, with no reference to a particular time zone or time of day.
A date
logical type annotates an Avro int
, where the int stores the number of days from the unix epoch, 1 January 1970 (ISO calendar).
The following schema represents a date:
{ "type": "int", "logicalType": "date" }
The time-millis
logical type represents a time of day, with no reference to a particular calendar, time zone or date, with a precision of one millisecond.
A time-millis
logical type annotates an Avro int
, where the int stores the number of milliseconds after midnight, 00:00:00.000.
The time-micros
logical type represents a time of day, with no reference to a particular calendar, time zone or date, with a precision of one microsecond.
A time-micros
logical type annotates an Avro long
, where the long stores the number of microseconds after midnight, 00:00:00.000000.
The timestamp-{millis,micros,nanos}
logical type represents an instant on the global timeline, independent of a particular time zone or calendar. Upon reading a value back, we can only reconstruct the instant, but not the original representation. In practice, such timestamps are typically displayed to users in their local time zones, therefore they may be displayed differently depending on the execution environment.
timestamp-millis
: logical type annotates an Avro long
, where the long stores the number of milliseconds from the unix epoch, 1 January 1970 00:00:00.000.timestamp-micros
: logical type annotates an Avro long
, where the long stores the number of microseconds from the unix epoch, 1 January 1970 00:00:00.000000.timestamp-nanos
: logical type annotates an Avro long
, where the long stores the number of nanoseconds from the unix epoch, 1 January 1970 00:00:00.000000000.Example: Given an event at noon local time (12:00) on January 1, 2000, in Helsinki where the local time was two hours east of UTC (UTC+2). The timestamp is first shifted to UTC 2000-01-01T10:00:00 and that is then converted to Avro long 946720800000 (milliseconds) and written.
The local-timestamp-{millis,micros,nanos}
logical type represents a timestamp in a local timezone, regardless of what specific time zone is considered local.
local-timestamp-millis
: logical type annotates an Avro long
, where the long stores the number of milliseconds, from 1 January 1970 00:00:00.000.local-timestamp-micros
: logical type annotates an Avro long
, where the long stores the number of microseconds, from 1 January 1970 00:00:00.000000.local-timestamp-nanos
: logical type annotates an Avro long
, where the long stores the number of nanoseconds, from 1 January 1970 00:00:00.000000000.Example: Given an event at noon local time (12:00) on January 1, 2000, in Helsinki where the local time was two hours east of UTC (UTC+2), and then converted to Avro long 946684800000 (milliseconds) and then written.
The duration
logical type represents an amount of time defined by a number of months, days and milliseconds. This is not equivalent to a number of milliseconds, because, depending on the moment in time from which the duration is measured, the number of days in the month and number of milliseconds in a day may differ. Other standard periods such as years, quarters, hours and minutes can be expressed through these basic periods.
A duration
logical type annotates Avro fixed
type of size 12, which stores three little-endian unsigned integers that represent durations at different granularities of time. The first stores a number in months, the second stores a number in days, and the third stores a number in milliseconds.