blob: bfd8ea7b4f33c30f44dab05c7c6151f1665cd060 [file] [view]
---
title: Xlang Serialization
sidebar_position: 2
id: xlang
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
Xlang is Fory's default object serialization mode. It uses one portable binary format across Java,
Python, C++, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. Each runtime's
[Basic Serialization](#runtime-guides) page owns its API and model examples; this page explains the
rules that peers must share.
Read [Core Concepts](core-concepts.md) first for the object graph, schema, reference, and
polymorphism concepts shared by xlang and native modes.
## Overview
Use xlang serialization when bytes cross runtime boundaries, including polyglot services, data
pipelines, and frontend/backend communication. It provides:
- Direct serialization of native language models without requiring an IDL.
- Coordinated numeric or named identities for application types.
- Compatible schema evolution for independently deployed peers.
- Optional shared-reference and circular-reference preservation.
- Polymorphic values when every concrete type has a portable mapping.
- Out-of-band buffers for large binary and numeric data where the runtime supports them.
Use [Native Serialization](native.md) instead when every writer and reader uses the same supported
runtime and the object graph needs language-specific behavior such as Java serialization hooks or
Python pickle-compatible objects.
### Supported Runtimes
| Runtime | Package or target | Modes |
| --------------------- | ------------------------------------------ | ------------ |
| Java | `org.apache.fory:fory-core` | xlang/native |
| Python | `pyfory` | xlang/native |
| C++ | Fory C++ CMake or Bazel target | xlang/native |
| Go | `github.com/apache/fory/go/fory` | xlang/native |
| Rust | `fory` crate | xlang/native |
| JavaScript/TypeScript | `@apache-fory/core` | xlang |
| C# | `Apache.Fory` | xlang |
| Swift | `Fory` Swift Package Manager target | xlang |
| Dart | `fory` package | xlang |
| Scala | `org.apache.fory:fory-scala` | xlang/native |
| Kotlin | `org.apache.fory:fory-kotlin` and Java API | xlang/native |
### First Cross-Language Round Trip
Register the same logical type identity and compatible fields on every peer. The following example
uses a shared type name.
Rust producer:
```rust
use fory::{Fory, ForyStruct};
#[derive(ForyStruct)]
struct Person {
name: String,
age: i32,
}
let mut fory = Fory::builder().xlang(true).build();
fory.register_by_name::<Person>("example.Person").unwrap();
let person = Person {
name: "Alice".to_string(),
age: 30,
};
let bytes = fory.serialize(&person).unwrap();
```
Python consumer:
```python
from dataclasses import dataclass
import pyfory
@dataclass
class Person:
name: str
age: pyfory.Int32
fory = pyfory.Fory(xlang=True)
fory.register_type(Person, name="example.Person")
person = fory.deserialize(bytes_from_rust)
```
Although xlang is the default, examples often select it explicitly so the transport contract is
visible in application code.
### Native Models or Fory IDL
| Approach | Use when |
| --------------------------------- | ---------------------------------------------------------------------- |
| Native language models | The contract is small and teams want to start without a compiler step |
| [Fory IDL](../compiler/index.md) | Many messages or teams need one schema and generated models |
| Native first, then migrate to IDL | A small contract is becoming a long-lived, multi-team service boundary |
A minimal Fory IDL message looks like this:
```protobuf
package example;
message Person {
string name = 1;
int32 age = 2;
optional string email = 3;
}
```
Generate the required runtime targets with `foryc`; generated models use consistent field and type
metadata across those targets.
## Type System and Type Identity
### Built-in and Custom Types
Primitive numeric values, strings, binary values, temporal values, lists, sets, maps, dense numeric
arrays, enums, structs, and unions have shared xlang schemas. Built-in values do not require user
registration. Application structs, enums, unions, and extension types require a coordinated
identity.
The normative [type mapping](../specification/xlang_type_mapping.md) defines the exact host-language
carrier for every xlang type. Important cases include:
- Python uses markers such as `pyfory.Int32`, `pyfory.Float16`, and `pyfory.BFloat16` when the native
Python type does not express the required width.
- Java, Dart, and other runtimes use annotations or schema metadata where one host type can represent
multiple xlang types.
- Reduced-precision `float16` and `bfloat16` values and dense arrays use runtime-specific carriers.
- `list<T>` and dense `array<T>` are distinct schemas. In compatible mode, a direct struct field may
adapt between a list and dense bool/numeric array when the element domain is compatible and the
actual list contains no unrepresentable null or reference-tracked element.
Use the mapping specification instead of inferring compatibility from similar host-language names.
### Coordinate Type Identity
Every peer must register a custom type with either the same numeric ID or the same namespace and type
name. Numeric IDs produce smaller metadata; names are easier to coordinate across independently
owned services. Do not register one peer by ID and another by name for the same contract.
Registration must happen before the first root serialization or deserialization operation. Keep a
small contract registry or use generated Fory IDL modules when multiple teams own the peers.
### Static and Dynamic Fields
A statically known field uses its declared serializer without writing a concrete runtime type. A
dynamic field carries enough type information to select the concrete registered type. Dynamic
metadata is needed for interfaces, abstract types, trait objects, and other polymorphic positions;
it is unnecessary for primitives and exact final types.
| Runtime | Dynamic field model |
| ------- | ----------------------------------------------------------------- |
| Java | `@ForyField(dynamic = ...)` controls automatic or forced metadata |
| Python | `pyfory.field(dynamic=...)` controls object-field metadata |
| C++ | `fory::F(...).dynamic(...)` overrides automatic detection |
| Go | Interface fields express dynamic values |
| Rust | Trait-object carriers express dynamic values |
Writing dynamic metadata costs space and type-resolution work. Disable it only when the field can
never contain another concrete type. Exact annotation and registration examples belong to each
runtime's schema metadata, type registration, and polymorphism pages.
## Nullability and Reference Tracking
Nullability and reference tracking solve different problems:
| Concern | Purpose |
| ------------------ | ------------------------------------------------------------- |
| Nullability | Allows a field or value position to contain no value |
| Reference tracking | Preserves repeated object identity and supports object cycles |
The wire framing is defined by the
[xlang serialization specification](../specification/xlang_serialization_spec.md). Applications
should configure the semantic behavior through runtime APIs rather than depend on flag values.
### Nullability
Xlang struct fields are non-nullable by default. Non-nullable fields are smaller and make required
data explicit. Nullable or optional carriers include Java boxed/annotated values, Python
`Optional[T]`, C++ `std::optional<T>`, Go pointers, Rust `Option<T>`, and Scala `Option[T]`.
Keep corresponding fields consistently nullable across peers. In same-schema mode, changing
nullability changes the schema and is incompatible. Compatible mode supports the documented
nullable and scalar adaptations, but a remote null still cannot be materialized into a local type
that has no valid null or missing-value behavior.
### Shared and Circular References
Enable reference tracking when the graph contains the same object more than once or contains a
cycle. Leave it disabled for value-shaped data to avoid identity-table overhead.
```java
import org.apache.fory.Fory;
import org.apache.fory.annotation.Ref;
public class Node {
public String value;
@Ref public Node next;
}
Node first = new Node();
Node second = new Node();
first.next = second;
second.next = first;
Fory fory = Fory.builder()
.withXlang(true)
.withRefTracking(true)
.build();
```
Global reference tracking enables the runtime mechanism; field metadata selects which positions
participate. Common field-level controls are Java and Scala `@Ref`, Go `fory:"ref"` tags, Rust
`#[fory(ref = true)]`, and C++ smart-pointer or `fory::F().ref()` metadata. Consult the runtime guide
because default tracking differs by carrier and language.
Reference support also follows the host ownership model. For example, Rust can preserve supported
shared-reference carriers, while cycles require representable ownership and weak-reference shapes.
## Polymorphism
Xlang polymorphism preserves a value's concrete registered type when its declared field, collection
element, or root type is broader. Every receiving peer must:
1. Register the same concrete type identity.
2. Provide a compatible field schema for that concrete type.
3. Mark or model the position as dynamic when the runtime cannot infer it.
4. Use a concrete type that has a portable xlang mapping.
Host-language inheritance alone does not make a type portable. If a shape has no xlang mapping, use
the runtime's native mode for same-language traffic or define a portable model. See the runtime
polymorphism pages for interfaces, trait objects, unions, and generated-code syntax.
## Schema Evolution
Compatible mode is the xlang default. It carries schema metadata so independently deployed readers
can tolerate supported field additions, removals, reordering, and documented compatible type
adaptations.
Keep compatible mode when peers may deploy separately. Select same-schema mode only when every
reader and writer uses the same:
- Type identity and field IDs or names.
- Field types and nested generic shapes.
- Nullability and reference metadata.
- Polymorphic alternatives.
Same-schema mode reduces metadata and payload size, but any mismatch can produce a schema hash or
type error. Generated Fory IDL models make exact coordination easier when all peers are released
together. Normative compatibility behavior lives in the
[xlang serialization specification](../specification/xlang_serialization_spec.md).
## Zero-Copy Serialization
Some runtimes can move large binary or numeric buffers out of the main serialized byte stream. This
avoids copying those buffers into one contiguous payload.
The transport flow is:
1. Serialize the object graph and collect selected buffer objects through a callback.
2. Send the main metadata bytes and the collected buffers separately.
3. Provide the buffers in the same order during deserialization.
Java:
```java
Collection<BufferObject> objects = new ArrayList<>();
byte[] metadata = fory.serialize(value, object -> !objects.add(object));
List<MemoryBuffer> buffers = objects.stream()
.map(BufferObject::toBuffer)
.toList();
Object decoded = fory.deserialize(metadata, buffers);
```
Python:
```python
objects = []
metadata = fory.serialize(value, buffer_callback=objects.append)
buffers = [obj.to_buffer() for obj in objects]
decoded = fory.deserialize(metadata, buffers=buffers)
```
Go exposes the equivalent callback-buffer flow through its serialization and buffer APIs. Use the
runtime documentation for the current method names and supported buffer carriers.
Out-of-band serialization helps when buffers are large and the transport can send them without an
additional copy. For small arrays, callback and multi-buffer transport overhead may cost more than
copying. The application owns buffer ordering, lifetime, and transport framing. See
[Python Out-of-Band Serialization](python/out-of-band.md) for Python and NumPy details.
## Troubleshooting
| Symptom | Likely cause | Resolution |
| ---------------------------------------- | --------------------------------------------------- | -------------------------------------------------------------- |
| Type is not registered | Registration missing or performed too late | Register every custom type before the first root operation |
| Type ID or name mismatch | Peers use different identities | Use the same numeric ID or the same namespace and type name |
| Integer overflow or float precision loss | Host carriers use different numeric widths | Follow the type mapping and use explicit width metadata |
| Fields decode incorrectly | Field IDs, names, or types differ | Align field metadata or regenerate all peers from the same IDL |
| Stack overflow on a cyclic graph | Reference tracking is disabled | Enable global and field-level reference tracking |
| Shared objects become duplicates | The value position does not track references | Enable reference tracking for that carrier or field |
| Unsupported host type | The type has no portable xlang representation | Use a portable model or native mode for same-language traffic |
| Schema/hash mismatch | Same-schema peers have different schemas | Align every peer or restore compatible mode |
| Failure after an upgrade | Peers run incompatible protocol versions | Align supported Fory versions and review release notes |
| Payload rejected immediately | One peer wrote native bytes and another reads xlang | Keep all peers on xlang for a cross-language contract |
### Diagnostic Checklist
1. Confirm that every peer uses xlang mode and a mutually supported Fory version.
2. Compare the registered type identity, field IDs or names, numeric widths, nullability, and
reference metadata.
3. Reproduce a same-runtime round trip before testing the cross-runtime direction.
4. Test both directions for every language pair used in production.
5. Reduce the value to one type and field, then add fields back until the mismatch appears.
6. Inspect the runtime-specific troubleshooting page for generated-code, platform, or API errors.
When diagnosing binary layout, use the specifications and runtime debug facilities. Do not treat a
hex dump or internal flag value as a stable application API.
## Runtime Guides
- [Java](java/basic-serialization.md#cross-language-interoperability)
- [Python](python/basic-serialization.md#cross-language-interoperability)
- [C++](cpp/basic-serialization.md#cross-language-interoperability)
- [Go](go/basic-serialization.md#cross-language-interoperability)
- [Rust](rust/basic-serialization.md#cross-language-interoperability)
- [JavaScript/TypeScript](javascript/basic-serialization.md#cross-language-interoperability)
- [C#](csharp/basic-serialization.md#cross-language-interoperability)
- [Swift](swift/basic-serialization.md#cross-language-interoperability)
- [Dart](dart/basic-serialization.md#cross-language-interoperability)
- [Scala](scala/basic-serialization.md#cross-language-interoperability)
- [Kotlin](kotlin/basic-serialization.md#cross-language-interoperability)
## Related Documentation
- [Xlang Serialization Format](../specification/xlang_serialization_spec.md) — normative wire format
- [Xlang Type Mapping](../specification/xlang_type_mapping.md) — exact runtime carrier mappings
- [Fory IDL and Compiler](../compiler/index.md) — schema-first models and code generation
- [Getting Started](../start/index.md) — installation and first serialization for each runtime
- [Row Format](../row-format/index.md) — random-access analytical rows for trusted data
## Operational Best Practices
1. Coordinate one type identity and field contract across every peer.
2. Keep compatible mode unless all readers and writers deploy the same schema together.
3. Enable reference tracking only for identity-bearing or cyclic graphs.
4. Reuse configured Fory instances instead of rebuilding them for each operation.
5. Validate every production language pair in both directions before deployment.
6. Prefer Fory IDL once a contract spans many messages, services, or independently owned teams.