blob: e749d37da43ede033c28cf0089d66c6196020a5a [file] [view]
---
title: Xlang Implementation Guide
sidebar_position: 10
id: xlang_implementation_guide
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
## Overview
This guide describes the current xlang implementation ownership model used by
the xlang runtimes.
The wire format is defined by
[Xlang Serialization Spec](xlang_serialization_spec.md). This document is about
service boundaries, operation flow, and internal ownership. New language implementations do not
need the same class names, but they should preserve the same control flow:
- root operations stay on the `Fory` facade
- nested payload work stays on explicit read and write contexts
- type metadata stays in the type resolver layer
- serializers stay payload-focused
When this guide conflicts with the wire-format specification, follow
`docs/specification/xlang_serialization_spec.md`. When it conflicts with a
language-specific implementation detail, follow the current implementation code for
that language.
## Source Of Truth
Use these sources in this order:
1. `docs/specification/xlang_serialization_spec.md`
2. the current implementation for the language
3. cross-language tests under `integration_tests/`
For Dart, the implementation shape is centered on:
- `Fory`
- `WriteContext`
- `ReadContext`
- `RefWriter`
- `RefReader`
- `TypeResolver`
- `StructSerializer`
## Implementation Ownership Model
### `Fory` is the root-operation facade
`Fory` owns the reusable services for one Fory instance.
In Dart, `Fory` owns exactly four reusable members:
- `Buffer`
- `WriteContext`
- `ReadContext`
- `TypeResolver`
In Java, `Fory` also owns instance-local services such as `JITContext` and
`CopyContext`, but the ownership rule is the same: `Fory` is the root facade,
not the place where nested serializers do their work.
`Fory` is responsible for:
- preparing the shared buffer for root operations
- writing and reading the root xlang header bitmap
- delegating nested value encoding to `WriteContext`
- delegating nested value decoding to `ReadContext`
- owning registration through `TypeResolver`
- resetting operation-local context state in a top-level `finally`
Nested serializers must not call back into root `serialize(...)` or
`deserialize(...)` entry points.
### `WriteContext` and `ReadContext` hold operation-local state
`WriteContext` and `ReadContext` are prepared by `Fory` for one root operation
and reset by `Fory` in a `finally` block before reuse.
`prepare(...)` should only bind the active buffer and root-operation inputs.
`reset()` should clear operation-local mutable state.
That operation-local state includes:
- the current buffer
- the active `RefWriter` or `RefReader`
- meta-string state
- shared type-definition state
- operation-local scratch state keyed by identity
- logical object-graph depth
Generated and hand-written serializers should treat these contexts as the only
source of operation-local services. Serializers must not keep ambient instance
state in thread locals, globals, or serializer instance fields.
### `WriteContext`
`WriteContext` owns all write-side per-operation state:
- current `Buffer`
- `RefWriter`
- `MetaStringWriter`
- shared TypeDef write state
- root `trackRef` mode
- recursion depth and limits
It exposes one-shot primitive helpers such as:
- `writeBool`
- `writeInt32`
- `writeVarUInt32`
These helpers are convenience methods. Serializers that perform repeated
primitive IO should cache `final buffer = context.buffer;` and call buffer
methods directly.
### `ReadContext`
`ReadContext` owns all read-side per-operation state:
- current `Buffer`
- `RefReader`
- `MetaStringReader`
- shared TypeDef read state
- recursion depth and limits
It exposes matching one-shot primitive helpers such as:
- `readBool`
- `readInt32`
- `readVarUInt32`
Generated struct serializers call `context.reference(value)` immediately after
constructing the target instance so back-references can resolve to that object.
## Reference Tracking
Reference handling is split behind two explicit services:
- `RefWriter` writes null, ref, and new-value markers and remembers previously
written objects by identity.
- `RefReader` decodes those markers, reserves read reference IDs, and resolves
previously materialized objects.
The xlang ref markers are:
- `NULL_FLAG (-3)`
- `REF_FLAG (-2)`
- `NOT_NULL_VALUE_FLAG (-1)`
- `REF_VALUE_FLAG (0)`
Key behavior:
- basic values never use ref tracking
- field metadata controls ref behavior inside generated structs
- root `trackRef` is only for top-level graphs and container roots with no
field metadata
- serializers that allocate an object before all nested reads complete must bind
that object early with `context.reference(...)`
## Type Resolution
`TypeResolver` owns:
- built-in type resolution
- registration by numeric id or by `namespace + typeName`
- serializer lookup
- struct metadata lookup
- type metadata encoding and decoding
- canonical encoded meta strings for package names, type names, and field names
- encoded-name lookup for named type resolution
- wire type decisions for struct, compatible struct, enum, ext, and union forms
In Java xlang mode the concrete implementation is `XtypeResolver`. In Dart the
same ownership stays behind the internal `TypeResolver`.
Serializers do not resolve class metadata themselves. They ask the current
context to read or write nested values, and the context delegates type work to
`TypeResolver`.
### External-type serialization ownership
A language binding that cannot attach Fory behavior directly to every
application value type may separate a serializer provider from its target
value.
The ownership split is:
- the serializer provider owns static serialization behavior and, for an
external structural serializer, the local schema declaration
- the target owns runtime values, host type identity, storage size, and
dynamic downcasts
- the value serializer owns target bodies, complete root values, defaults,
value type identity, root/dynamic type information, and value-level
polymorphism
- the internal field codec extends the value serializer for the same target
and owns `FieldType`, field null/reference framing, field-capacity hints,
remote field metadata, compatible-field composition, and recursive carrier
field schemas
- one carrier implementation owns the body, allocation, insertion, and
reference algorithms reused by root serializers and field codecs
- a custom serializer owns allocations inside its opaque body and must perform the
corresponding readable-byte, policy, and graph-memory checks before
allocation
- `Fory` owns root framing and operation setup/reset
- `TypeResolver` owns registration and dynamic lookup
#### C# generated structural serializers
C# uses one target-keyed source-generation path for ordinary and external
structural serializers. `ForyStructAttribute` is non-inherited: every
first-party class that participates in a serializable hierarchy carries a
direct annotation. An external declaration supplies the equivalent contract
for an unmodifiable target.
For one concrete ordinary class, compilation-level hierarchy discovery
produces two independent immutable outputs:
- one flattened wire-member set, used by field ordering, schema hashes,
`TypeMeta`, and generated reads and writes; and
- one shallow-storage model, used only by graph-memory accounting.
The wire set is assembled base-first from declaration-owned generated
descriptors. Property override chains collapse to one logical slot before the
complete set is validated and sorted by the protocol comparator. Hidden
members retain their exact declaring type. A generated child never calls a
parent serializer body and never encodes a base object as a nested value.
Each inheritable ordinary class publishes a target-keyed compiler contract
containing:
- its exact target;
- its exact directly declared wire-member count;
- one descriptor on each deterministic accessor for a directly declared wire
member: fields use a writable `ref` accessor, while properties use a getter
plus a matching setter; and
- a public static readonly cumulative `HierarchyShallowBytes` value.
The provider's shallow value is the immediate parent provider value plus only
the current class's directly declared physical instance fields. A sealed
concrete serializer uses the same cumulative expression privately and does not
publish a provider marker, descriptors, or hierarchy value. The concrete
serializer adds object self storage once. Properties never directly add
storage, while private, readonly, compiler-generated, and non-wire instance
fields do. Referenced child compilations consume only an accessible provider
contract; they do not import, enumerate, or reconstruct private parent fields.
Provider-only targets emit a static provider; a concrete non-sealed
serializer carries the same contract without a second type or forwarding path.
An internal provider is available only to assemblies granted normal C#
accessibility, such as through `InternalsVisibleTo`; an inaccessible or
extern-alias-only contract does not own another compilation's hierarchy.
The generator emits ordinary direct member access whenever C# accessibility
allows it. A provider publishes an accessor when a referenced child must reach
declaration-owned state. The accessor signature preserves the member's CLR
type and nullability metadata. Missing or ambiguous providers fail generation.
An abstract ordinary class emits only its generated hierarchy provider and does not
create a serializer instance or registration. Its property descriptors may
publish unresolved abstract override slots; a concrete descendant must supply
the callable implementation. Concrete classes require legal parameterless
construction and retain the existing allocate-before-children and reference
publication order.
`ForyStructAttribute` and `ForyEnumAttribute` carry an optional `Target` type.
A local non-generic abstract class supplies an external structural declaration,
while an empty non-generic static class selects an external enum target:
```csharp
[ForyStruct(Target = typeof(ThirdParty.User))]
internal abstract class UserSerializer
{
[ForyField(
1,
TargetDeclaringType = typeof(ThirdParty.User),
TargetMemberName = "<Name>k__BackingField")]
public abstract string Name { get; }
}
[ForyEnum(Target = typeof(ThirdParty.Status))]
internal static class StatusSerializer
{
}
```
External declarations are compile-time generator input only. They are never
instantiated, reflected over by the runtime, registered, reference-published,
or used as wire identities. Runtime type positions, construction, `TypeInfo`,
metadata, reference publication, generated factory keys, roots, fields,
dynamic values, and carriers use the target type.
An external member may bind a visible same-name field or property. For external
class targets, setting `TargetDeclaringType` and `TargetMemberName` instead
declares one exact field on the target or a non-`object` ancestor. An exact wire mapping
also supplies physical storage; `Ignore = true` supplies only shallow storage.
External struct targets support visible member mappings only. Unmapped visible
public instance fields are added to external class shallow storage once. The
generator never discovers private fields from a referenced assembly.
`BaseOnly = true` makes an external class declaration the terminal provider for
a complete third-party hierarchy prefix. It may list exact target and
target-ancestor fields and publishes no standalone factory or registration. An
ordinary child consumes this provider exactly as it consumes an ordinary
parent provider. A `BaseOnly` target may be abstract or nonconstructible
because only the concrete ordinary child is materialized.
Exact private mappings are version-pinned package ABI declarations. Wire
accessors fail with the CLR missing-field error if the target ABI changes;
there is no reflection or alternate-member fallback. On .NET 8, the generator
rejects private wire access whose declaring owner or signature is generic.
Visible closed-generic members and explicit storage-only field mappings remain
supported for class targets. An inaccessible pointer field cannot be
distinguished from fixed-buffer storage without importing private layout, so an
exact private pointer mapping is rejected.
Standalone external structural targets require an accessible concrete class
or struct, legal parameterless construction, and writable declared wire state.
Constructor-only, factory-only, readonly, init-only, converted, and
custom-wire shapes use a custom `Serializer<T>`. Explicit nullability must
match when target metadata is annotated; otherwise the declaration supplies
schema nullability.
Every generated ordinary or external struct uses
`TypeResolver.RegisterGeneratedStruct<T, TSerializer>(bool evolving)` to carry
generator-owned `Evolving` into target `TypeInfo`. Generated enums and unions
use `TypeResolver.RegisterGenerated<T, TSerializer>()`. Abstract ordinary and
`BaseOnly` providers do not register. Multiple generated owners for one target
are rejected during generation or deterministically on the cold
cross-assembly factory-registration path. Custom serializer replacement keeps
the resolver's normal target rules.
C# carrier composition remains target-based. The resolver recursively binds
`Nullable<T>`, one-dimensional `T[]`, `List<T>`, `LinkedList<T>`, `Queue<T>`,
`Stack<T>`, `HashSet<T>`, `SortedSet<T>`, `ImmutableHashSet<T>`,
`Dictionary<TKey, TValue>`, `SortedDictionary<TKey, TValue>`,
`SortedList<TKey, TValue>`, `ConcurrentDictionary<TKey, TValue>`, and
`NullableKeyDictionary<TKey, TValue>`. Ordinary, external, and custom
serializers use the same carrier bodies. There is no hierarchy runtime lookup,
provider object, callback, schema tree, per-element dispatch, or additional
value allocation.
Dynamic `object` values and unions resolve concrete target types through
`TypeResolver`. Arbitrary statically typed interface or base-class
polymorphism remains unsupported. Flat ordinary and external generated hot
bodies retain the same work and allocation shape apart from target/member
metadata tokens; hierarchy composition is static initialization and
compile-time metadata work.
Rust names these serializer operation boundaries explicitly:
- `Serializer::write` and `Serializer::read` process a complete value,
including the requested reference and type-information envelopes.
- `Serializer::write_data` and `Serializer::read_data` process only the target
body.
- `write_with_type_info` and `read_with_type_info` remain complete-value
operations over already-resolved value metadata.
Rust represents immutable value-serializer properties with five associated
constants on `Serializer`:
- `IS_OPTIONAL` means the selected value shape carries Option semantics;
- `IS_POLYMORPHIC` means its concrete target is selected from the runtime
value;
- `IS_SHARED_REF` means it uses the existing shared-reference wire behavior;
- `IS_WRAPPER` means it is a Fory-owned wrapper serializer without an
independent registration identity; and
- `REQUIRES_SCOPED_ACCESS` means inspecting or using the contained dynamic
value requires a borrow, lock, or weak upgrade.
These are type-level value properties and must fold through monomorphization.
`Codec<T>` inherits them through `Serializer<Target = T>` and does not
redeclare them. `SerializerCodec<S>` forwards them from `S`. The
value-dependent `is_none(value)` and `dynamic_type_id(value)` operations
remain functions. `is_none` means only `Option::None`, including transparent
Option propagation; weak-target expiry remains a separate weak-access result.
`Serializer` has no field API or field-schema argument. In particular, it does
not expose `FieldType`, field-compatible reads, declared-field generic state,
field null/reference policy, or field encoding selection. Rust's internal
`Codec<T>` extends `Serializer<Target = T>` and owns those field operations.
The leaf `SerializerCodec<S>` implements value behavior by forwarding to `S`
and implements field behavior itself; it never calls a field hook on `S`.
Its value-capacity hint is forwarded unchanged. Generated fields and
field-mode carrier bodies use the codec-owned field-capacity hint, which adds
or forwards field framing without changing root or value composition.
Serializer-provider identity is a host implementation detail and is never
encoded. External structural serializers use the same STRUCT, ENUM, or UNION
metadata and value format as an equivalent directly supported target. Custom
serializers that are not the runtime's canonical implementation of an existing
built-in use EXT or NAMED_EXT. Serializer-provider separation does not replace
runtime-owned built-in mappings.
Static generated fields and serializer-selected roots should dispatch directly
to the serializer selected by their schema. A Rust field `with = S` selects the
exact field node and requires `S::Target` to equal the declared field type.
This accepts an ordinary, external structural, custom, or carrier serializer.
For example, `with = VecSerializer<UserSerializer>` selects the structural
`Vec<User>` field node, while `list(element(with = UserSerializer))` selects
the child node recursively. Transparent fields select their exact carrier
serializer, such as `OptionSerializer<UserSerializer>`. Field codegen lowers
both forms recursively to carrier codecs.
Rust procedural macros cannot resolve type aliases or renamed imports. Because
the design has no associated codec mapping trait or runtime fallback, every
carrier constructor in a schema-bearing derived field's declared Rust type and
every Fory-owned carrier constructor in its `with` tree must use its canonical
terminal name, optionally qualified. Leaf serializer aliases, root carrier
aliases, and carrier aliases used only by a skipped field's value-level default
remain valid. Derive recognizes the complete audited carrier syntax and lowers
unknown types only as leaf serializers. The leaf adapter rejects an aliased
non-wrapper carrier during cold field-schema construction using its existing
wire category, before resolver publication. An aliased Fory-owned wrapper
cannot be registered independently and therefore fails the ordinary
required-provider lookup. The leaf adapter does not consume `IS_WRAPPER`,
inspect runtime type names, or add a value-path branch.
When a root is a transparent,
collection, fixed-array, map, or heterogeneous tuple composition containing
selected external children, the binding should expose binding-owned,
carrier-specific static serializers parameterized recursively by child
serializers. For example, a vector carrier serializer over `S` has the target
vector of `S::Target`, a map carrier serializer over `KS` and `VS` has the
target map of `KS::Target` to `VS::Target`, and an arity-N tuple carrier
serializer has the target tuple formed
positionally from every `Si::Target`.
Root carrier composition recursively composes value serializers. Field
composition recursively composes codecs. They are intentionally different
compile-time type trees: a root carrier must not construct a field-codec tree
or request or synthesize `FieldType`. Root carrier reads may use only direct or
value-`TypeInfo` child state; remote `FieldType` belongs exclusively to the
field-codec entrance. Each carrier implementation provides `Serializer`
behavior when its children implement `Serializer`, and field `Codec` behavior
only when its children implement `Codec`. Both layers call one carrier body,
allocation, insertion, and reference implementation. They must not copy
collection, map, reference, holder, fixed-array, tuple, or compatible-read
algorithms. A metadata-only adapter that still calls a whole carrier
serializer is not external-child support.
Rust keeps an external structural serializer's generated `write_data` body
non-inlined. Recursive carrier composition would otherwise duplicate that body
into each list, map, tuple, and wrapper monomorph. This is a direct
successful-path function boundary, not a cold path, and it adds no runtime
selection, callback, or allocation. Self-owned generated serializers retain
the ordinary compiler inlining heuristic.
A transparent reference carrier consumes only its own null or reference
envelope. If compatible mode places user-type metadata before the selected
child body, the child must still consume that metadata and use its remote
schema. If the containing schema instead declares a recursive carrier child,
the child receives that declared field metadata directly. Implementations must
not discard either metadata form or read the carrier envelope twice.
Carrier serializers are not registered user types: they preserve the carrier's
standard wire kind, have no resolver entry or dynamic harness, and any
registration belongs to a selected user-type child when that child is reached.
Carrier delegation must include every existing canonical specialization rather
than forcing a generic collection shape. For example, a Rust vector carrier
serializer over the canonical `i32` serializer retains `INT32_ARRAY`, one over
the canonical `u8` serializer retains BINARY, and one over an external
structural or custom serializer uses LIST. A nested vector retains the selected
child representation in root bytes; the equivalent field-codec tree retains it
in recursive `FieldType`.
For Rust, the audited carrier serializer surface is exhaustive:
- `OptionSerializer<S>`, `BoxSerializer<S>`, `RcSerializer<S>`,
`ArcSerializer<S>`, Fory `RcWeakSerializer<S>`/`ArcWeakSerializer<S>`,
`RefCellSerializer<S>`, and `MutexSerializer<S>`;
- `VecSerializer<S>`, `VecDequeSerializer<S>`,
`LinkedListSerializer<S>`, `HashSetSerializer<S>`,
`BTreeSetSerializer<S>`, `BinaryHeapSerializer<S>`, and
`ArraySerializer<S, N>`;
- `HashMapSerializer<KS, VS>` and `BTreeMapSerializer<KS, VS>`;
- `Tuple1Serializer<S0>` through
`Tuple22Serializer<S0, ..., S21>`.
Every carrier serializer target is formed recursively from child serializer
targets. Each child can be an ordinary serializer targeting itself, an external
structural serializer, a custom serializer, or another carrier serializer, and
all four forms enter the same carrier body implementation. `Tuple1Serializer` through
`Tuple22Serializer` and matching
arity-specific codecs are macro-generated because Rust has no variadic
generics; no public serializer-list trait or runtime tuple descriptor is
introduced. Unit `()` and `PhantomData<T>` have no serialized child and require
no serializer composition. Erased Any/application-trait carriers retain their
dynamic registered-target owner rather than masquerading as static child
serializers.
`Cell<T>` is not in the current Rust serializer surface: derive only recognizes
its Send/Sync properties. There is no `Cell` serializer or codec, so this
feature must not invent `CellSerializer<S>` or treat `Cell` as `RefCell`.
Likewise, standard-library weak pointers are not aliases for Fory's weak
carriers.
For Swift, `Serializer` follows the same exact-target ownership boundary with
an associated `Target`. A self-provided structural or custom serializer uses
`Target == Self`; a separately provided structural or custom serializer names
another target type. Serializer operations are static and accept or return
`Target`. Fory never instantiates a serializer object, and generated external
structural code reads target properties and constructs the target directly.
Static selection follows provider ownership uniformly. A target that itself
conforms to `Serializer` with `Target == Self` is selected implicitly at roots,
generated fields, optionals, arrays, sets, and dictionaries. This includes an
external type with an intentional retroactive conformance. A serializer type
whose `Target` is another type must be selected explicitly at every static node
where it is required. Registration does not infer that serializer, even if it
is the only registered provider for the target, because Swift cannot
reverse-infer a unique `S: Serializer` from `S.Target == T`.
A retroactive conformance is process-global. `@retroactive` acknowledges
Swift's ownership warning but does not make duplicate `(Target, Protocol)`
conformances safe. Applications may use it when they intentionally own the
single global binding; public libraries should generally use a separate
serializer so applications can choose the implementation explicitly.
Swift `StructSerializer` covers every structural registration category.
Ordinary and external `@ForyStruct`, `@ForyEnum`, and `@ForyUnion` expansions
all conform; custom EXT serializers do not.
Swift's doc-hidden `FieldCodec` extends value serialization for the same exact
target. It owns `FieldType`, recursive field generics, field null/reference
policy, compatible-field reads, scalar conversion, and annotation-selected
field encodings. `Serializer` has no `FieldType`, compatible-field hook, or
declared-generics argument. `FieldCodec` alone carries the
`hasDeclaredChildren` mode that preserves canonical collection and map header
decisions when enclosing field metadata owns the recursive child schema.
`SerializerCodec<S>` is the generated leaf adapter and has target `S.Target`.
Because `FieldCodec` extends `Serializer`, its value-level half must remain a
valid root serializer, but root code cannot observe any field-only operation.
In non-compatible mode, exact nonoptional selected leaves with no
type-information envelope call `S.readData` when no reference envelope is
required; tracked reference targets continue through `S.read`. Compatible or
recursively retained metadata remains owned by `SerializerCodec<S>` and enters
the selected serializer with that scope intact.
Numeric field codecs delegate that value-level half to the canonical numeric
serializer, and packed-array field codecs delegate it to the canonical LIST
carrier. Only their field operations use fixed, tagged, or packed field
representations. Root eligibility therefore adds no alternate numeric or
packed-array wire mapping.
Swift `Serializer.isWrapper` is a doc-hidden value-level property used only to
reject Fory-owned transparent wrappers as independent EXT registrations.
`OptionalSerializer` sets it; collection carriers do not. A custom serializer
does not acquire wrapper status from target spelling and may own an independent
opaque carrier body. Target-identity conflicts prevent it from replacing a
seeded canonical dynamic builtin.
The exhaustive Swift recursive carrier serializer surface is:
- `OptionalSerializer<S>` with target `S.Target?`;
- `ArraySerializer<S>` with target `[S.Target]` and LIST wire shape;
- `SetSerializer<S>` with target `Set<S.Target>` where `S.Target: Hashable`;
- `DictionarySerializer<KS, VS>` with target
`[KS.Target: VS.Target]` where `KS.Target: Hashable`.
Each carrier serializer conditionally provides field-codec behavior when its
children are field codecs. Root composition therefore contains serializers,
while field composition contains recursively lowered field codecs, and both
call the same carrier body, allocation, insertion, reference, and compatible
implementation. Ordinary `Optional`, `Array`, `Set`, and `Dictionary`
conformances delegate to the same owners under exact self-target constraints.
Those constraints apply equally to user-declared and retroactively conforming
external children. Carrier serializers are zero-state and unregistered.
The transparent Swift `OptionalSerializer` has no independent field-metadata
identity. Its field-codec metadata scope delegates recursively to the wrapped
field codec, including nonnull reads, so accepted remote child metadata remains
owned by the selected leaf or nested carrier.
Swift `Array` is LIST for statically selected roots and carrier roots, including
`[Int32]`. `@ArrayField` is a separate field-only dense bool/numeric selection,
and exact primitive arrays hidden in dynamic `Any` use their canonical dynamic
packed-array mapping. Only a canonical primitive field codec can select
the packed form; a noncanonical selected child is rejected rather than gaining
a mapping from its target syntax. `Data` remains the BINARY leaf.
Swift has no supported generic tuple, fixed-length array, cell, box, weak,
reference-holder, `ContiguousArray`, `ArraySlice`, result, range, deque,
`NSSet`, or `NSDictionary` serializer. External-child composition must not
invent those carriers. `AnyHashable`, `UnknownCase`, and `ByteBuffer` are a
dynamic key holder, union value holder, and transport owner respectively, not
recursive static carriers.
Swift external structural declarations use the structural macros:
```swift
@ForyStruct(target: ThirdParty.User.self)
struct UserSerializer {
var name: String
var age: UInt32
}
```
Generated ordinary value declarations use `Target = Self`. Because Swift
rejects a nested `Target = Self` type alias in a class, an ordinary generated
class names the declaring class explicitly; both forms express the same exact
self-target relationship.
Equivalent target selection applies to `@ForyEnum` and `@ForyUnion`. A value
schema declaration targets a value type and uses direct labeled construction.
A class schema declaration targets a class, allocates the final target through
an accessible zero-argument initializer, publishes it before reading child
fields, and assigns accessible mutable properties. Generated Swift enforces
the class target's `AnyObject` constraint. Swift has no negative generic
constraint for the inverse case, so cold registration validation rejects a
value-schema declaration that targets a class before publishing metadata. An
inaccessible, immutable, invariant-bearing, or non-exhaustive target requires a
custom serializer; the implementation must not use reflection, unsafe layout
access, unavailable-overload tricks, schema mirror values, conversion wrappers,
or builders as a fallback.
Swift macros cannot inspect another type's stored layout. An external class's
shallow graph-memory formula therefore uses its declaration fields only.
`@ForyField(ignore: true)` adds a budget-only declaration field without schema,
target access, construction, or wire code. Applications must use this form for
substantial omitted storage.
An external structural union requires the target to expose a lossless
`unknown(UnknownCase)` case. A dependency-free target module may expose a
generic unknown payload and let the application select its `UnknownCase`
specialization; a target may also use Fory's carrier directly. Fory does not
convert another module's unknown representation. A third-party union without
this shape requires a custom serializer and does not claim structural-union
wire equivalence.
An ordinary Swift generated field recursively selects a self-provided declared
type without annotation. Swift field selection uses
`@ForyField(with: S.self)` for one exact declared node and `.with(S.self)`
inside `ForyFieldType` list, set, map, and union payload nodes when selecting a
separate serializer or another deliberate override. A selected optional or
whole collection node names its exact carrier serializer. Canonical
whole-carrier syntax is recursively lowered to the same field-codec tree as the
structural field DSL. The compiler enforces that the selected serializer target
equals the declared field node.
Swift root selection uses a serializer metatype:
```swift
fory.serialize(user, with: UserSerializer.self)
fory.deserialize(bytes, with: UserSerializer.self)
fory.serialize(users, with: ArraySerializer<UserSerializer>.self)
```
The `Data`, append-to-`Data`, and `ByteBuffer` forms share one root framing and
the reusable contexts. Ordinary roots require `T.Target == T` and
delegate to the same selected-serializer helper. This admits every
self-provider, including an intentional retroactive external conformance, but
does not infer a separate serializer from registration. There are no parallel
serializer-selection aliases or application-declared structural container
schemas. The root facade retains its existing module boundary and inlining
policy; static specialization belongs to serializer, generated-code, and
carrier owners below it. Do not expose resolver or reusable-context state as
`@usableFromInline` merely to force the complete root flow into clients.
Selected Swift read roots carry an inferred result generic constrained by
`S.Target == T`. The call remains `deserialize(..., with: S.self)`, but the
same-type generic lets the caller provide concrete target metadata and avoids
an associated-target lookup and dynamic result-storage setup in every
unspecialized root call.
Generated value-struct `readData` owns target construction directly. When an
external structural serializer has recursively selected carrier fields, that
owner may be out of line to control code size, but a private large-value
returning helper must not sit between `readData` and the final target result
buffer. Generated class readers retain a private helper because the class
allocation owner must publish a reserved reference before reading children.
Swift `TypeResolver` indexes one immutable `TypeInfo` by both serializer
identity and concrete target identity. Static schema and explicit selection use
serializer identity; dynamic writes use target identity; wire reads use the
numeric ID or name. All directions share one writer, exact reader, compatible
reader, metadata, and registration owner. Public registration accepts the
selected structural or custom serializer through the ID or name API. It rejects
carrier, dynamic, builtin, and field codec identities before publication.
Swift arbitrary application protocol existentials use a zero-state
`DynamicSerializer<T>`. Dynamic writes resolve the registered concrete target,
downcast once to its serializer's exact target, and invoke the shared
`TypeInfo` harness. Dynamic reads materialize the final concrete target once
and cast it to the requested existential. Normal target registration is the
allowlist, so Swift needs no marker protocol, protocol-specific registry,
closed target-list macro, protocol wire identity, serializer value, or wrapper
collection. Protocol roots and root carriers select the dynamic serializer
explicitly, for example `DynamicSerializer<any Animal>` and
`ArraySerializer<DynamicSerializer<any Animal>>`. Swift retains direct `Any`
and `AnyObject` root overloads for source compatibility. Those overloads
forward to the same selected-serializer roots with `DynamicSerializer<Any>` or
`DynamicSerializer<AnyObject>` and add no parallel codec, framing, lookup, or
allocation path. Their append-to-Data and ByteBuffer forms follow the same
rule. Because Swift permits every value to convert to `Any`, a concrete
non-self-serializing value may enter registered dynamic-target lookup through
the `Any` overload. Explicit `with:` selection names a separate static
serializer when required. Swift exposes no unconstrained generic dynamic root,
`any Serializer` root, or specialized heterogeneous-container root overload.
`DynamicSerializer<T>` overrides complete-value reference handling. It resolves
the concrete `TypeInfo` before deciding whether the target is a reference and
uses conservative `isRefType == true` only for the outer dynamic field shape.
The outer dynamic serializer owns nullability and type information. Its
`TypeInfo` harness invokes body serialization for a value target and the
concrete complete-value reference envelope for a class target. Dynamic slot
accounting uses the declared existential layout rather than treating every
value as a reference. `AnyObject` reads reject value-typed metadata before the
cast so Swift cannot allocate a bridge object. An arbitrary nonoptional
protocol has no synthetic default; optional protocol values compose through
`OptionalSerializer`.
An exact generated field whose selected codec has UNKNOWN static identity
still writes and reads the concrete `TypeInfo`; carrier headers own this
information for dynamic children and do not duplicate it. Dynamic map chunks
retain the established key-type/value-type/key-body/value-body order and reuse
the two resolved `TypeInfo` entries without another target lookup. A retained
dynamic `TypeInfo` is scoped by the serializer that selected it, so
`AnyHashable` and `DynamicSerializer<T>` share one implementation without
substituting each other's scope identity.
Swift rejects zero-sized non-null MAP chunks because they cannot advance the
decoded entry count. A one-null entry encodes its non-null side as a complete
field value: its reference envelope when present, then undeclared `TypeInfo`,
then its body. The MAP writer, reader, and compatible-field skipper use that
complete-field order rather than the shared type-prefix order of non-null
chunks. Static and dynamic MAP branches use the same ordering and validation.
Dynamic Any and protocol paths operate directly on final target values and
containers. Static composition never enters target lookup; dynamic lookup
occurs only at an explicit dynamic boundary.
Swift heterogeneous dynamic collections use their exact supported target
shapes: `[Any]`, `[String: Any]`, `[Int32: Any]`, and `[AnyHashable: Any]`.
Assigning a homogeneous list or map to `Any` does not erase its concrete target
identity and must not trigger a converted collection or a second collection
codec. Homogeneous lists and maps use their ordinary or explicitly selected
carrier serializer. Preseeded exact primitive arrays retain their packed
dynamic mapping. A null dynamic key in `[AnyHashable: Any]` materializes as
`AnyHashable(ForyAnyNullValue())`.
When a carrier retains remote child metadata for a user-type field codec, the
leaf codec enters the selected serializer without another envelope so its exact
or compatible reader sees that retained `TypeInfo`. Builtin leaf codecs read
their field bodies directly.
Swift supports only the xlang wire mode. A known `@ForyUnion` case has zero or
one associated value, and multiple logical fields use an explicit struct
payload. There is no Swift-native multi-field enum encoding to preserve.
Rust tuple field metadata selects sparse zero-based positions, while
unmentioned positions use ordinary serializers. It lowers to the
arity-specific tuple codec, while the root carrier recursively composes tuple
serializers; both use the same arity-specific tuple body. The existing tuple wire
contract remains unchanged: native non-compatible mode uses direct
heterogeneous position bodies, while compatible native and xlang modes use
the existing heterogeneous LIST form. Its existing UNKNOWN generic
`FieldType` shape remains unchanged, and no serializer type or position index is
encoded.
Rust's primitive collection selection must have one owner shared by ordinary
types, carrier serializers, and derive. Type ID, body encoding, reserved space,
field metadata, and compatible reads cannot use independent tables. This
includes canonical `u8` BINARY and the existing `isize`/`usize` dense-array
types. The private primitive-array/carrier owner derives the parent kind from
the child's scalar `static_type_id()`, the exact Rust child target, and
the carrier mode. Scalar serializers and codecs declare only scalar behavior
and scalar wire IDs; neither public serializer contracts nor the generic codec
contract exposes a parent-array kind. The same private mapping validates unsafe
bulk copies and supplies compatible LIST/array element metadata.
Rust 1.70 cannot select a codec type from an associated const, so the existing
Vec and fixed-array carrier implementation types are the single owners of both
primitive and object bodies. The Vec implementation carries two established schema choices as compile-time
consts. `STRUCTURAL_LIST` preserves unannotated and explicit `list(...)`
generated fields as LIST even for canonical primitive children. Ordinary roots,
Vec carrier serializers, `#[fory(bytes)]`, and `#[fory(array)]` disable it so
the carrier can consume the validated canonical child kind. `DENSE_ARRAY` is true
only for explicit `#[fory(array)]`; it maps canonical `u8` BINARY to
`UINT8_ARRAY`. It is false for roots, carrier serializers, LIST fields, and
`#[fory(bytes)]`. Consequently, an unannotated `Vec<i32>` field remains
`LIST<VARINT32>`, an explicit fixed-element list remains `LIST<INT32>`, and
ordinary or serializer-selected primitive Vec roots retain their dense-array or
BINARY representation. An external structural or custom serializer targeting a
primitive remains an object LIST child because its serializer does not expose the
canonical scalar wire ID. Derive may validate the explicit primitive category
but does not map Rust types to wire IDs. Inline scalar-ID and exact-target
checks must fold after monomorphization. Derive always uses the same unified
core carrier implementation and maintains no ordinary-composition AST
primitive table.
Registration is access-driven by the owning serializer or field codec. A
selected user serializer must be registered before schema construction or value
processing accesses its registered identity or registration-backed metadata.
An absent Option, empty collection or map, empty weak value, zero-length fixed
array, or equivalent recursive branch may complete without child registration
when its owning path makes no such access. A declared-type body path invokes
its statically selected serializer without a registration lookup; if a
containing schema supplies that declaration, its metadata construction already
owns the one required registration/mismatch check. A heterogeneous tuple whose
existing outer `FieldType` does not declare positions instead reaches any
required child registration through its ordinary per-position type-metadata
operation. A binding must not add an eager recursive root validation pass,
reached-body check, selector tree, composed-target lookup, per-element
serializer dispatch, allocation, callback, or hot-path branch to change that
behavior. An exact custom serializer for a whole container remains a separate
opaque EXT/NAMED_EXT choice. It owns registered dynamic target identity, while
unregistered carrier serializers continue to own explicit static structural
composition.
Carrier serializers have no independent registered identity. Structural
registration requires the existing structural serializer contract and matching
STRUCT/ENUM/UNION category. Custom registration requires an independent
EXT/NAMED_EXT serializer and rejects `IS_WRAPPER`. Option, Box, Rc, Arc,
RcWeak, ArcWeak, RefCell, and Mutex carrier serializers set this property
independently of their child's category. Lists, sets, maps, fixed arrays, and
tuples keep it false and are rejected through their own wire category. A
custom serializer targeting one of the same Rust shapes also keeps it false
because it owns an independent opaque EXT body. Private built-in registration
validates only its expected internal type ID. These semantic checks use the
existing serializer contracts and wire categories. Custom EXT registration is
the only runtime consumer of `IS_WRAPPER`.
Dynamic values should resolve by the concrete target identity. When a runtime
needs both directions, its serializer-provider-to-type-info and target-to-type-info
indexes must point to the same immutable registration metadata and serializer
harness rather than creating parallel metadata or serialization paths.
The internal context and resolver entry points must name the direction; Rust
uses `write_provider_type_info`/`get_provider_type_info` for static schema
identity and `write_target_type_info`/`get_target_type_info` for dynamic value
identity. An ambiguous lookup must not probe one map and fall back to the other.
Wire reads still resolve their encoded ID or name to that same metadata.
When an existing homogeneous LIST/SET or MAP header emits one dynamically
selected concrete type, that header owner resolves and validates the target
once and retains the resolved registration metadata for the wire chunk. Each
body borrows that exact metadata to invoke the existing dynamic harness; it
must not repeat target lookup or clone a reference-counted metadata handle per
element or entry. Heterogeneous chunks retain their existing per-value metadata
path. This handoff adds no wire field, runtime serializer instance, callback,
schema tree, cache, or static-path branch.
Dynamic target inspection is fallible and represents an absent value without a
sentinel target identity. A LIST/SET owner may pre-inspect a child when
`!C::IS_POLYMORPHIC || !C::REQUIRES_SCOPED_ACCESS`; this caller-local decision
must not become another serializer capability. A polymorphic child that needs
a `RefCell` borrow, `Mutex` lock, or weak upgrade skips target/null
pre-inspection and writes each value through the existing heterogeneous path
under one holder access. A non-polymorphic nullable holder preserves the
existing LIST/SET null-header scan, then performs one body access without
repeating null inspection. MAP metadata and null flags for both sides precede
either body, so a nullable or access-constrained polymorphic MAP holder
performs one short null/target inspection, releases the borrow, guard, or
upgraded owner, and later performs its normal body access. Implementations must
not retain that access across the other map side, stage body bytes, allocate a
prepared value, invoke a callback, or change MAP wire order. A weak wrapper
does not keep its target alive; an operation requiring one coherent
observation must retain its upgraded strong owner for that owned operation.
If closed polymorphic membership needs the host target identity after a wire
ID/name lookup, that shared harness or registration metadata must retain an
optional target identity. Registered local behavior supplies it; a remote-only
stub supplies none and must enter the missing-registration error before
membership checks, serializer calls, or allocation. Do not scan a reverse map,
invent a sentinel identity, or add a second metadata cache.
Serializer code must not materialize a serializer provider value or a
structural mirror value. Generated structural reads construct the target directly. Dynamic
materializers allocate the requested final owner once and use target size for
memory accounting.
Fallible serializer defaults that can materialize values must receive the
active read state or context. A field codec uses its inherited serializer
default rather than defining a second default API. The concrete owner reserves
graph memory before allocating; null, missing-compatible-field, and
skipped-field paths must not bypass the normal allocation budget merely because
they consume no body bytes.
Compatible structural reads remain in the normal struct-compatibility owner.
The checked metadata cache remains the sole owner of accepted remote metadata;
serializer dispatch must not add another validation marker, metadata cache, or
exact-schema decision.
## Root Frame Responsibilities
Every root payload starts with a one-byte bitmap written and read by `Fory`
itself, not by serializers.
Current xlang root bits:
| Bit | Meaning |
| --- | -------------------------- |
| `0` | null root payload |
| `1` | xlang payload |
| `2` | out-of-band buffers in use |
Keep the root bitmap separate from per-object ref markers:
- the root bitmap describes the whole payload
- ref flags describe one nested value at a time
## Serialization Flow
### Root write path
The current root write flow is:
1. `Fory.serialize(...)` or `serializeTo(...)` prepares the target buffer.
2. `Fory` calls `writeContext.prepare(...)`.
3. `Fory` writes the root bitmap.
4. `Fory` delegates the root object to `WriteContext`.
5. `writeContext.reset()` runs in `finally`.
For a non-null root value, `WriteContext.writeRootValue(...)` performs:
1. ref/null framing
2. type metadata write
3. payload write
Payload serializers are responsible only for the payload of their type. They do
not write the root bitmap and they do not own registration or type-header
encoding.
### Nested writes use `WriteContext`
Important rules:
- nested serializers must use `WriteContext` helpers such as `writeRef(...)`,
`writeNonRef(...)`, and container helpers when they need ref handling or type
metadata
- repeated primitive writes should go directly through the buffer
- nested serializer flow should stay straight-line; do not add internal
`try/finally` blocks just to clean per-operation state
- top-level `Fory.serialize(...)` owns the operation reset `finally`
## Deserialization Flow
### Root read path
The current root read flow mirrors the write flow:
1. `Fory.deserialize(...)` or `deserializeFrom(...)` reads the root bitmap.
2. null roots return immediately.
3. `Fory` validates xlang mode and other root framing requirements.
4. `Fory` calls `readContext.prepare(...)`.
5. `Fory` delegates to `ReadContext`.
6. `readContext.reset()` runs in `finally`.
### `ReadContext` owns ref reservation and payload materialization
`ReadContext.readRef()` performs the normal xlang read sequence:
1. consume the next ref marker
2. return `null` or a back-reference immediately when appropriate
3. reserve a fresh read ref id for new reference-tracked values
4. read type metadata
5. read the payload
6. bind the reserved read ref id to the completed object
Primitive and string-like hot paths should read directly from the buffer;
complex payloads delegate to the resolved serializer.
### Stream And Buffer Byte Reads
Implementations must keep byte availability in the byte owner layer while
keeping string, binary, primitive-array, compression, and collection semantics in
serializers.
The required byte-owner primitive for allocation-before-read checks is a
readability check such as `checkReadableBytes(byteCount)`. Implementations do
not need additional generic read-context methods for this design. After the
readability check succeeds, serializers use their existing local buffer read,
copy, or decode paths.
The readability check is a byte operation only. It must not decode strings,
primitive-array element counts, compression modes, or collection capacity
policy.
For large byte-counted values, every implementation should call the byte-owner
readability check before allocating a variable-length result. This applies to
binary values, strings, decimal or metadata bodies, and primitive wire arrays
whose encoded body is measured in bytes. For multi-byte primitive wire arrays,
compare the encoded byte count, not only the logical element count, with the
readable bytes.
1. Validate the encoded byte count in the serializer. For fixed-width primitive
arrays, check overflow and element alignment before allocation, such as
`wireByteCount % elementByteWidth == 0`, then derive the logical element
count from the encoded byte count.
2. Call `checkReadableBytes(wireByteCount)` unconditionally before allocating
the variable-length result. Buffer-backed inputs normally return from this
check with only a bounds comparison. Stream-backed inputs use the same call;
the byte owner handles the fast path when enough bytes are already buffered
and otherwise fills the read buffer until the requested encoded body is
readable or an input error is recorded.
3. After readability is proven, allocate the final value once and copy or decode
from the current readable buffer into the final result.
`checkReadableBytes` is not an `ensureCapacity(wireByteCount)` operation. In
stream mode it may end with the byte owner holding the full encoded body in its
read buffer, but it must grow that buffer as bytes are successfully read from
the stream. It should grow from current proven buffer capacity, such as by
doubling current capacity, and cap only when that bounded growth step reaches
the immediate target. A byte owner may use an owner-local availability signal as
a one-shot growth hint when the stream implementation itself is caller-owned
trusted code; if that hint is absent or insufficient, it must fall back to
bounded growth from already buffered bytes. It must not reserve the
attacker-declared length before input bytes or an owner-local growth hint
justify that intermediate buffer capacity. The stream slow path may pay one
extra intermediate buffer copy; this is preferable to serializer-local chunk
accumulation and repeated final-container growth.
For byte-counted values, the serializer should not duplicate the byte owner's
fast-path branch by testing `availableBytes()` before calling
`checkReadableBytes`. Keeping that branch in the byte owner gives every language
the same correctness rule and keeps serializer hot paths focused on their own
wire semantics.
For primitive wire arrays:
- Compare and prove the encoded wire byte count, not only the logical element
count.
- Keep compression, bit-packing, byte-order conversion, and other primitive
array encoding semantics in the serializer. `checkReadableBytes` only proves
that the encoded bytes are present.
- For compressed or transformed bodies, the serializer must still validate the
decoded length and encoding-specific metadata before allocating or returning
the final value.
The common serializer shape is:
```text
wireByteCount = readVarUint32()
elementWidth = primitiveWireElementWidth(kind)
validate wireByteCount and element alignment
elementCount = wireByteCount / elementWidth
ctx.checkReadableBytes(wireByteCount)
result = allocatePrimitiveResult(elementCount)
copy or decode wireByteCount bytes from the current readable buffer into result
advance the reader index by wireByteCount
return result
```
Byte values are the `elementWidth == 1` specialization of the same policy. In
that case the serializer shape is:
```text
byteCount = readVarUint32()
ctx.checkReadableBytes(byteCount)
result = allocateBytes(byteCount)
copy byteCount bytes from the current readable buffer into result
advance the reader index by byteCount
return result
```
This policy avoids three inefficient implementation shapes:
- allocating the complete final contiguous value before the encoded body is
readable
- growing or repeatedly copying the final result container on stream slow paths
- adding serializer-local chunk buffers when the byte owner can prove
readability once and expose a normal buffered read
Scratch buffers remain appropriate when the target representation is not a
direct byte target, such as string transcoding, compression, byte-order
conversion that is not performed in place, bit-packed values, or runtimes whose
stream API cannot read into a caller-provided target.
For fixed-width primitive arrays, the final result must not become visible to
callers until the exact encoded byte count has been read successfully.
For list, set, map, and other container readers, the declared logical element
count is not an encoded byte count, so serializers must still own all element,
chunk, nullability, reference, and type-dispatch semantics. It is still the
right allocation proof for count-based preallocation: after validating a
non-empty count and reading any serializer-owned header or type metadata that
precedes allocation, call `checkReadableBytes(logicalCount)` before allocating,
reserving backing capacity, or size-hinting from that count. The byte owner
handles buffer versus stream readiness; the container serializer then allocates
with the declared count and reads elements through its normal owner path.
This check is not a full container-body validation. It only prevents a small or
truncated input from causing a large count-based preallocation. Chunk sizes,
duplicate keys, element value semantics, and protocol strictness remain owned by
the container/map serializer and should be validated only when they protect a
real owner invariant.
Materializing readers should also reserve a root-operation estimated graph
memory budget before allocation or size hinting. The budget state belongs to
`ReadContext` or the equivalent root read state, not to ambient thread-local
state. Root facades set or reset the per-operation budget only; they must not
pre-reserve root type or root self bytes. `maxGraphMemoryBytes` defaults to a
fixed `128 MiB`; positive configuration overrides the default; explicit
non-positive configuration is invalid and must be rejected when the runtime is
created. Do not derive this budget from root input size, and do not add dynamic
stream bytes-read accounting for this budget.
Because the budget is fixed per root, read state should not mirror the
configured maximum into a second active-limit field. Use the existing
configuration, or one configured maximum field when the config is not otherwise
available, plus the mutable remaining budget.
Read context or equivalent read state owns only raw byte reservation. It must
not expose counted arithmetic helpers or collection, map, array, struct, or
object semantic reservation APIs. Concrete serializers and generated serializer
owners compute the storage constants and formulas for the owner path they
allocate, including counted-byte overflow checks.
Read state must not grow non-memory-budget APIs for this feature, including
ref-publication controls, temporary-owner controls, serializer-owner controls,
conversion helpers, or APIs that encode the kind of value being materialized.
Concrete serializers and generated serializers own those decisions.
The budget is an approximate gate for materialized graph owners, mainly
collections, maps, arrays, structs, and objects. It does not measure exact heap
bytes, and actual process memory can be higher. Reserve self storage exactly
once at the owner that stores or allocates the value. Root facades reset the
budget only and must not reserve root value storage. Reference-backed
containers, maps, sets, and
object/reference arrays reserve nonzero owner self cost plus reference slots;
each referenced heap owner then reserves its own shallow self cost when
materialized. Inline/value containers reserve element storage; inline/value maps
reserve key plus value storage; pointer, box, and dynamic materialization owners
reserve the heap or boxed storage they allocate. Value serializers, including
root and generated struct/product read paths, do not reserve their own self
storage. Struct/record/POJO/tuple, compatible, generated, and dynamic object
owners reserve a nonzero shallow self cost plus shallow field storage only in
reference-object runtimes or dynamic/boxed materialization paths.
Parents must not recursively include child object, collection, map, string,
binary, or primitive dense-array contents. Skip enum/union as separate owners and
skip dedicated string, binary, primitive scalar, primitive array, and primitive
dense-array leaf owners, but do not skip general inline-value containers such as
vectors or lists of value objects. If reference slot size is not cheap or
reliable to query, use a 4-byte reference slot. Native runtimes may use
conservative lower-bound estimates instead of guessing non-portable object,
container, allocator, table, node, entry, or debug-layout details. Reject
arithmetic overflow before budget comparison or allocation, and keep the
existing `checkReadableBytes` proof before backing
allocation or capacity reservation.
Skipped leaf owners must still be gated by remaining input bytes. If unread
bytes are insufficient for a string, binary value, primitive scalar, primitive
array, or primitive dense array, the runtime must not read or create that leaf
value.
For TypeDef or TypeMeta bodies, first prove that the encoded metadata body bytes
are readable through the byte owner. Field-list allocation should happen after
that body readability check and should not use a separate small initial-capacity
cap as a security rule.
Implementations should also bound received metadata bodies and struct field
lists on the cold metadata parse path. `maxTypeMetaBytes` limits one encoded
TypeDef or TypeMeta body, excluding the 8-byte header and any extended-size
varint, and is checked before copying or decompressing that body.
`maxTypeFields` limits the number of fields declared by one received struct
metadata body and is checked before reserving or allocating the field list.
These limits are runtime resource controls; they do not change wire encoding,
type identity, dynamic loading, unknown-type behavior, deserialization policy,
or schema-evolution semantics. Metadata cache hits and generated field readers
remain hot paths and must not add work for these limits.
Remote schema-version limits belong to the same cold metadata owner path.
Header cache hits must skip the remaining metadata body and return cached
metadata without schema-limit checks, hash revalidation, allocation, or policy
work. On a header miss, keep the handling in one concrete owner path: prove and
read the metadata body bytes, validate the body against its header, validate
field counts, resolve the type through the existing registration and
deserialization-policy checks, compare exact local metadata by original encoded
bytes when applicable, check schema-version limits for non-local remote
metadata, build the required read state, publish to the persistent metadata
cache, and then record the schema count. Failed or incompatible metadata must
not publish to the persistent cache and must not consume schema-version counts.
Remote metadata whose encoded bytes exactly match the local registered metadata
may use the local metadata without consuming the remote schema-version limit,
after the existing type and deserialization-policy checks for selecting that
local type have run. This exact-local bypass is not struct-only; it also applies
to named enum, ext, and union metadata when those metadata bodies are present and
match the local encoded bytes. Pure id-based enum, ext, and typed-union values
do not carry TypeDef or TypeMeta bodies and must stay on the normal type-id plus
user-type-id path. Compatible named enum, ext, and union metadata normally has
one version, but it still counts against accepted remote metadata totals when it
is sent as shared metadata and does not exactly match local metadata.
`maxTypeFields` applies only to struct field lists.
The exact-local candidate must be derived inside the metadata owner path from
the decoded metadata identity: `userTypeId` for id-registered metadata, or
`(namespace, typeName)` for name-registered metadata. Do not thread extra
expected-type parameters through read callers solely for this check. This rule
applies to every runtime. Java and Python may lazy-build the local encoded
metadata only after this identity lookup selects a local class and the existing
class, registration, and deserialization-policy checks for that class have run.
When a statically declared compatible named enum, ext, or union field reads
shared metadata, the decoded metadata must match the declared type id,
namespace, and type name before the metadata owner publishes it to the
persistent cache or records a schema count. Already accepted header or reference
cache hits still skip the body and must not rerun body-hash, schema-limit, or
registration checks, but the field reader must not treat metadata for a
different declared named type as the current field's metadata.
Skip paths do not need to materialize skipped values. Existing byte-skip
operations should consume any available buffered prefix first, then skip or drop
remaining stream bytes in bounded steps.
### Nested reads use `ReadContext`
Important rules:
- serializers that allocate the result object early must call
`context.reference(obj)` before reading nested children that may refer back to
it
- nested serializer flow should stay straight-line; do not add internal
`try/finally` blocks just to restore operation-local state
- top-level `Fory.deserialize(...)` owns the operation reset `finally`
## Depth Tracking
`WriteContext` and `ReadContext` track logical object depth explicitly.
`increaseDepth()` enforces `Config.maxDepth`.
Depth should stay explicit on the contexts rather than relying on the native
call stack alone. At the same time, depth cleanup should not depend on nested
`try/finally` blocks throughout serializer code. Top-level context reset must be
able to recover operation-local state after failures.
## Struct Compatibility
Struct-specific schema/version framing and compatible-field layout belong in the
struct serializer layer, not on `Fory` and not on the public serializer API.
In Dart that internal owner is `StructSerializer`.
`StructSerializer` is responsible for:
- schema-hash framing when compatibility mode is off and version checks are on
- compatible-struct field remapping when compatibility mode is on
- caching compatible read layouts
- skipping unknown compatible fields
- passing compatible read layouts explicitly to generated serializers
- classifying matched compatible fields as exact direct reads, compatible
conversions, or remote-only skips before generated dispatch
When `Config.compatible` is enabled and the struct is marked evolving:
- the wire type uses the compatible struct form
- the writer emits shared TypeDef metadata
- reads map incoming fields by identifier and skip unknown fields
- generated serializers apply matched fields directly while preserving their own
object construction and default-value rules
- exact matched field schemas use the same direct read shape as same-schema
reads and must not receive remote compatible metadata
- matched scalar fields may use compatible scalar conversion only when the
layout has classified a remote/local top-level scalar pair as lossless
convertible and both field schemas have `trackingRef = false`
- compatible scalar conversion applies only to the immediate matched field.
Nested collection, array, map key, and map value schemas must not be accepted
by recursively applying scalar conversion to child schemas.
- direct top-level `list<T?>` to dense `array<T>` matched fields must be
classified as compatible when element domains match; the nullable element
schema bit alone is not a schema-pair rejection. Actual null element payloads
fail in the dense-array reader. Ref-tracked list-element framing is separate
and may remain rejected when the runtime cannot materialize it without
generic/reference paths.
When `compatible` is disabled and `checkStructVersion` is enabled:
- the writer emits the schema hash for struct payloads
- the read side checks that hash before reading fields
Compatible scalar conversion is owned by the compatible struct field reader or
the generated compatible layout action. Root facades, read/write contexts, type
resolvers, class resolvers, xlang type resolvers, and raw buffer utilities must
not expose public conversion APIs or carry conversion state. Resolvers may
provide field schema metadata for layout classification, but the conversion
decision and value adaptation stay with the serializer-owned compatible field
layout. Layout classification must reject top-level scalar conversions when
either matched schema has `trackingRef = true` and must reject same scalar type
pairs whose top-level `trackingRef` framing differs; converters must not add a
reference-table path for scalar mismatches. Recursive schema comparison inside
containers must reject scalar mismatches instead of reusing the top-level scalar
conversion matrix. Generated serializers should consume the classified layout
decision directly:
- source-generated serializers use the layout's matched-field dispatch key to
select exact direct field code, compatible conversion code, or skip code
- regenerated serializers may instead compile a remote-schema-specific
straight-line reader after classification, without a second outer matched-id
switch, when the generated source still has pure direct, pure conversion, and
explicit skip operations
- compatible scalar conversion cases must read the concrete remote wire scalar
selected by classification and compose only the required lossless conversion;
they must not call a generic runtime converter that redispatches by remote and
local scalar type IDs, field descriptors, field names, or schema eligibility
helpers
Same-schema readers with matching reference and null/optional framing must keep
direct scalar read paths without conversion branches or per-field conversion
objects. Same raw scalar types with different null/optional framing may still
use the compatible nullable/optional composition path when both fields are not
reference-tracked.
## Meta Strings And Shared Type Metadata
Two explicit pieces of state back xlang type metadata:
- `MetaStringWriter` and `MetaStringReader` deduplicate and decode namespace
and type-name strings
- shared TypeDef write/read state tracks announced TypeDef metadata
Ownership rules:
- canonical encoded names live in `TypeResolver`
- per-operation dynamic meta-string ids live on `MetaStringWriter` and
`MetaStringReader`
- shared type-definition tables are operation-local context state
## Enums In Xlang Mode
In xlang mode, enums are serialized by numeric tag, not by name.
In Java:
- the default tag is the declaration ordinal
- `@ForyEnumId` can override that with a stable explicit tag
- `serializeEnumByName(true)` affects native Java mode, not xlang mode
In C#, the enum's underlying numeric value is the xlang tag. Java peers for
sparse C# enums must declare matching `@ForyEnumId` values instead of relying
on declaration ordinals.
Other language implementations should preserve the same wire rule even if the configuration or
annotation surface differs.
A Rust data-carrying enum is an xlang union only when each known variant is unit
or carries exactly one alternative value and satisfies the union case rules.
Rust native mode (`xlang = false`) may additionally encode tuple or named
variants with multiple fields through its native enum format. Those
struct-style enum shapes have no implicit xlang mapping. Registration in xlang
mode must reject them on the cold schema/type-selection path before publishing
resolver state; generated code must not drop fields, synthesize an undeclared
variant struct, or fall back to EXT.
## Out-Of-Band Buffer Objects
Buffer-object handling follows the same split:
- one root bit advertises whether out-of-band buffers are in play
- nested buffer-object payloads still decide in-band vs out-of-band one value at
a time
- serializers use read/write context helpers rather than bypassing the context layer
## Code Generation
The normal Dart integration path is:
1. annotate structs with `@ForyStruct`
2. annotate field overrides with `@ForyField`
3. run `build_runner`
4. call the generated per-library helper, such as
`<InputFile>ForyModule.register(...)`, to bind private generated metadata and
register generated types
Generated code should emit:
- private serializer classes
- private metadata constants
- a public per-library registration helper that users call from application code
- private generated installation helpers that keep serializer factories private
The public helper should be a thin generated wrapper around the Fory
registration API, not a public global registry or a second unrelated
registration API family.
### Dart Ordinary Struct Inheritance
Ordinary Dart `ForyStruct` inheritance is a code-generation-time field
discovery, normalization, access, construction, and flattening change. It does
not redesign the runtime reference protocol.
For a concrete annotated child, the generator walks the instantiated
superclass and applied-mixin storage chain rather than only the child's direct
`element.fields`. Each layer's `InterfaceType.element.fields` exposes its
declared elements, including private declarations in another library. Dart
privacy controls which generated expression can access an element; it does not
control whether the element is discovered.
The storage collector:
1. visits the instantiated superclass;
2. visits actually applied mixins in application order;
3. visits the current class;
4. collects concrete instance storage declared by each layer exactly once.
It excludes `Object`, interfaces, mixin `on` constraints, abstract accessors,
static fields, external fields, and synthetic members that do not own storage.
A mixin slot is identified by its application site and declaring field, so
multiple applications cannot be collapsed by spelling or `baseElement`.
Every discovered storage field follows one pipeline:
```text
complete hierarchy discovery
-> declaration-owned @ForyField(ignore: true)
-> concrete-child ignoreInheritedPrivateFields policy
-> concrete generic substitution
-> direct or companion access resolution
-> constructor validation
-> one globally sorted child schema
```
`@ForyField(ignore: true)` is the declaration-owned, per-field omission.
After that check, a concrete child with
`ignoreInheritedPrivateFields: true` removes every private field declared by a
superclass or applied mixin, including same-library, cross-library, direct, and
transitive ancestors. It does not remove child-declared private fields or
inherited public fields. Both omission forms bypass substitution, access,
construction, wire identity, reference analysis, and codec work while their
physical slots remain in the concrete object's shallow graph-memory field
count. Any unresolved field that remains included is a generation error.
Ownership is fixed:
| Concern | Owner |
| -------------------------------- | ------------------------------------------------ |
| Field identity and annotations | Declaring storage field |
| Hierarchy discovery/substitution | Concrete-child generator |
| Inherited-private omission | Concrete annotated child |
| Cross-library private permission | Public boundary in the field's declaring library |
| Schema, sort, and codec | Concrete annotated child |
| Construction | Selected concrete-child generative constructor |
| Reference analysis/publication | Existing concrete-child serializer path |
| Graph-memory self charge | One concrete child object |
| External target field list | Explicit external serializer declaration |
The concrete-child omission option defaults to `false`, is not inherited from
ancestor annotations, and is valid only on an ordinary concrete declaration
that owns a flattened schema. It is invalid on external target declarations
and provider-only abstract, open-generic, or mixin boundaries. It is applied
after complete storage discovery and before generic substitution or access
resolution, so a matching field needs no direct access or companion even when
a companion exists.
For fields that remain included, public inherited fields and private inherited
fields declared in the child's library use direct generated access and require
no parent annotation. A private field declared in another library requires
`ForyStruct(exposePrivateFields: true)` on a public hierarchy boundary in that
field's declaring library. `exposePrivateFields` defaults to `false`, is
invalid with `ForyStruct.target`, and authorizes only provider-library access
generation. It does not enable discovery, alter same-library access, change
field inclusion, or let a consumer authorize another library's private state.
A public boundary may expose same-library private storage inherited from a
private class or mixin. If private fields come from several Dart libraries,
each declaring library must independently provide an opted-in public boundary
and visible companion. The nearest qualifying child-visible boundary in the
declaring library is selected.
The provider's `.fory.dart` part emits a public `@nodoc` typed static access
companion. It emits an exact getter for each exposed non-ignored private field,
a setter only for mutable storage, and no setter for `final` or `late final`
storage. The receiver is the public boundary type. Generic bounds and every
type nested in a public signature must be nameable outside the provider
library. The companion must not use `dynamic`, `Object?` bridge casts,
reflection, callbacks, runtime lookup, a parent serializer, or stored runtime
state. Companion generation is independent of every consumer child's
`ignoreInheritedPrivateFields` value. A concrete boundary may enable both
options: its own serializer applies the omission, while its provider companion
continues to expose the declaring library's eligible private storage.
The child source must have a direct import or re-export namespace that exposes
the public boundary and companion. Provider output must be generated and
published before a dependent package is built. Missing permission, a hidden or
ambiguous companion namespace, an unnameable signature, or dispatch that no
longer reaches the exact storage slot is a generation error. A child must also
validate the complete concrete hierarchy so field hiding below the boundary
cannot redirect a generated getter or setter to another slot.
Every included `final` or `late final` field must be initialized by a
statically proven identity flow:
```text
selected concrete-child constructor parameter
-> redirect or super parameter
-> exact storage field
```
Accepted edges are exact field formals, super formals, direct parameter
references in constructor field initializers, and direct parameter references
in redirecting or super-constructor arguments. Types must remain identical
after concrete generic substitution, including nullability. Calls, operators,
casts, null assertions, constants, constructor-body assignment, and matching
names without element identity are not proof. A declaration initializer on a
included final field is unsupported because the decoded value cannot own that
slot. There is no post-construction final write, reflection, or fallback.
Mutable fields connected to selected constructor parameters are initialized
once through the same exact identity flow. Remaining mutable fields require an
exact setter and are restored after construction. Required constructor
parameters must have one unambiguous field source. Optional named parameters
may be omitted; an omitted optional positional parameter cannot be followed by
a passed positional parameter. Constructor arguments and assignments are
matched by resolved storage-field identity rather than field-name strings. If
omission removes the only serialized source for a required parameter,
generation fails; the generator must not invent a value or relax identity
proof.
The concrete child owns one `GeneratedStructSchema`, one canonical field sort,
one serializer and descriptor cache, one reconstruction, one reference
publication path, and one graph-memory owner. Parent serializers are neither
nested nor invoked, and parent runtime registration is not required. A
separately annotated concrete parent has its own independently flattened
schema only for values of that exact type.
Included direct and inherited fields feed one normalized list into the
existing recursive reference analysis and `needsRootRef` calculation.
Included inherited `ref: true` and nested container metadata behave like
equivalent direct child fields; omitted fields do not enter that list.
Inheritance adds no reference state, `ReadContext` API, serializer signature,
call-contract change, runtime branch, slot, sentinel, callback, wrapper,
compatible-layout state, or parent reference owner. A failure shared by the
equivalent flat model is a separate reference-subsystem issue.
Java provides only the flattened-model comparison for this feature: its object
serializer extends one field-descriptor list across inheritance and adds no
inheritance-specific reference state. Java's existing reference contract uses
the `readRefIds` stack, real reference IDs, and its `-1` sentinel to associate
`reference(obj)` with the current serializer invocation. That contract differs
from Dart's current reference subsystem and is not a design to transplant as
part of Dart hierarchy support. Dart inheritance is required only to match its
own equivalent flat model.
Hierarchy traversal, substitution, access validation, and constructor proofs
run during generation. Existing flat serializers gain no runtime work, public
and same-library inherited fields emit the same direct operations as equivalent
flat fields, policy filtering emits no runtime check, and included
cross-library private fields add only an inlineable typed static companion
call. Generation must not introduce runtime hierarchy traversal, allocation,
callbacks, reflection, or parent dispatch.
The child's shallow graph-memory formula is:
```text
24 + 4 * actualConcreteStorageFieldCount
```
It counts every real inherited and ignored storage slot once and adds no parent
self charge.
Generated diagnostics must identify the concrete child, declaring field,
declaring library, and failed access or constructor path, then give an
actionable remedy. Typical remedies are adding
`exposePrivateFields: true` to a public owner-library boundary, importing its
generated companion, forwarding a constructor value unchanged, placing
`@ForyField(ignore: true)` on the field declaration, setting
`ignoreInheritedPrivateFields: true` on the concrete child when all private
ancestor state should be omitted, or using a custom serializer. Diagnostics
must state that hierarchy discovery is independent of cross-library access.
Changes to hierarchy storage, exposure boundaries, or
`ignoreInheritedPrivateFields` require regeneration of every affected
`.fory.dart` file. Compatible mode uses normal missing/unknown-field handling;
fixed-schema peers must change together. Implementations must not add a
special legacy reader, retain an alternative path limited to child
declarations, bind constructors only by name, use a late-final setter path,
fall back to another access model, or delegate to a parent serializer.
Acceptance requires that every hierarchy storage slot is either omitted by its
declaration, omitted by the concrete-child inherited-private policy, or
represented exactly once with a valid type, access path, wire identity, and
reconstruction path. Equivalent included flat and inherited models must
produce the same canonical schema, wire bytes, reference IDs, and round-trip
behavior. External target declarations remain explicit and unaffected.
### Dart External Structural Serializers
Dart external-type serialization extends `ForyStruct` with an optional
compile-time `target` and named generative `constructor`. The annotated
`abstract final` class is a schema declaration only. It has no runtime value or
registration identity.
The generator must analyze ordinary and external structs through one struct
model and one emitter. Private generated symbol names come from the declaration
name. Every runtime type position uses the target type: `Serializer<Target>`,
`GeneratedStructSchema<Target>`, read and write signatures, constructor calls,
schema `type`, and generated-module dispatch.
For each serialized declaration field, resolve an accessible target getter
with the same name and exact instantiated Dart type. Constructor parameters
and any post-construction setter must also match exactly. A public named
generative constructor is selected only when the annotation names it. Factory
constructors, abstract targets, open target types, and constructor-based
reference-tracked paths back to the target are rejected during generation.
The recursive check includes target elements, keys, and values nested in the
supported list, set, and map field metadata.
The external declaration's fields are the complete schema. It may explicitly
name an accessible property inherited by the target, but the generator does
not automatically scan the external target hierarchy for schema fields.
`exposePrivateFields` and `ignoreInheritedPrivateFields` are invalid on an
external declaration.
An external object's shallow graph-memory formula is the union of declaration
fields and public instance fields discovered on the target, its superclasses,
and applied mixins. A public target field represented by a declaration is
counted once. `@ForyField(ignore: true)` adds budget-only declaration storage
without target access, construction, metadata, or wire code.
Generated code reads getters and invokes target constructors or setters
directly. It must not allocate the declaration, copy values through an
intermediate object, invoke runtime callbacks, perform member-name lookup, or
branch on whether the struct is external. The existing generated struct,
registration, resolver, field, collection, map, compatible-read, and reference
paths remain the only runtime paths.
Registration is keyed by `Target` through the generated module and the existing
generated registration API. Direct roots, generated fields, dynamic values,
and recursive collection/map children resolve the same target registration.
Dart root collections retain their existing untyped outer runtime shapes.
## Directory Layout
Under each Dart package `lib/` tree, only one nested source layer is allowed.
Allowed:
- `lib/fory.dart`
- `lib/src/<file>.dart`
- `lib/src/<area>/<file>.dart`
Not allowed:
- `lib/src/<area>/<subarea>/<file>.dart`
## Serializer Design Rules For New Implementations
Any new xlang implementation should follow these rules even if its surface API looks
different:
1. Keep root operations on the `Fory` facade and nested payload work on
explicit read and write contexts.
2. Keep reference tracking behind dedicated read-side and write-side services
so the disabled path stays cheap.
3. Make serializers payload-only. Type metadata, registration, and root
framing belong to the `Fory` and type resolver layers.
4. Track per-operation state explicitly. Do not rely on ambient thread-local
instance state.
5. Reserve read reference IDs before materializing new objects, and bind
partially built objects as soon as a nested child may refer back to them.
6. Keep operation setup and operation cleanup separate. `prepare(...)` binds
the current operation inputs, and `reset()` clears operation-local state.
7. Preserve the separation between the root bitmap, per-object ref flags, type
headers, and payload bytes.
8. Keep internal naming in the serialization domain. Prefer words like
`serializer`, `binding`, and `layout`; avoid RPC-style terms such as
`session` or vague control-flow terms such as `plan`.
9. When the implementation language supports cold and no-inline annotations,
keep error, cache-miss, schema-mismatch, unsupported-capability, and other
cold entrances reachable from serializer hot paths out of hot-path
inlining. Do not mark successful dynamic dispatch cold.
10. After any xlang protocol or ownership change, run the cross-language test
matrix and update both this guide and
[Xlang Serialization Spec](xlang_serialization_spec.md).
## Validation
For Dart implementation changes, run at minimum:
```bash
cd dart
dart run build_runner build
dart analyze
dart test
```
For generated consumer coverage, also run:
```bash
cd dart/packages/fory-test
dart run build_runner build
dart test
```