blob: 991ed5b22c20b010e2053bb6efadf2b86266ba14 [file] [log] [blame] [view]
---
title: Field Configuration
sidebar_position: 5
id: field_configuration
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
This page explains how to configure field-level metadata for serialization.
## Overview
Apache Fory™ provides two ways to specify field-level metadata at compile time:
1. **`fory::field<>` template** - Inline metadata in struct definition
2. **`FORY_FIELD_TAGS` macro** - Non-invasive metadata added separately
These enable:
- **Tag IDs**: Assign compact numeric IDs for schema evolution
- **Nullability**: Mark pointer fields as nullable
- **Reference Tracking**: Enable reference tracking for shared pointers
## The fory::field Template
```cpp
template <typename T, int16_t Id, typename... Options>
class field;
```
### Template Parameters
| Parameter | Description |
| --------- | ------------------------------------------------ |
| `T` | The underlying field type |
| `Id` | Field tag ID (int16_t) for compact serialization |
| `Options` | Optional tags: `fory::nullable`, `fory::ref` |
### Basic Usage
```cpp
#include "fory/serialization/fory.h"
using namespace fory::serialization;
struct Person {
fory::field<std::string, 0> name;
fory::field<int32_t, 1> age;
fory::field<std::optional<std::string>, 2> nickname;
};
FORY_STRUCT(Person, name, age, nickname);
```
The `fory::field<>` wrapper is transparent - you can use it like the underlying type:
```cpp
Person person;
person.name = "Alice"; // Direct assignment
person.age = 30;
std::string n = person.name; // Implicit conversion
int a = person.age.get(); // Explicit get()
```
## Tag Types
### fory::nullable
Marks a smart pointer field as nullable (can be `nullptr`):
```cpp
struct Node {
fory::field<std::string, 0> name;
fory::field<std::shared_ptr<Node>, 1, fory::nullable> next; // Can be nullptr
};
FORY_STRUCT(Node, name, next);
```
**Valid for:** `std::shared_ptr<T>`, `std::unique_ptr<T>`
**Note:** For nullable primitives or strings, use `std::optional<T>` instead:
```cpp
// Correct: use std::optional for nullable primitives
fory::field<std::optional<int32_t>, 0> optional_value;
// Wrong: nullable is not allowed for primitives
// fory::field<int32_t, 0, fory::nullable> value; // Compile error!
```
### fory::not_null
Explicitly marks a pointer field as non-nullable. This is the default for smart pointers, but can be used for documentation:
```cpp
fory::field<std::shared_ptr<Data>, 0, fory::not_null> data; // Must not be nullptr
```
**Valid for:** `std::shared_ptr<T>`, `std::unique_ptr<T>`
### fory::ref
Enables reference tracking for shared pointer fields. When multiple fields reference the same object, it will be serialized once and shared:
```cpp
struct Graph {
fory::field<std::string, 0> name;
fory::field<std::shared_ptr<Graph>, 1, fory::ref> left; // Ref tracked
fory::field<std::shared_ptr<Graph>, 2, fory::ref> right; // Ref tracked
};
FORY_STRUCT(Graph, name, left, right);
```
**Valid for:** `std::shared_ptr<T>` only (requires shared ownership)
### fory::dynamic\<V\>
Controls whether type info is written for polymorphic smart pointer fields:
- `fory::dynamic<true>`: Force type info to be written (enable runtime subtype support)
- `fory::dynamic<false>`: Skip type info (use declared type directly, no dynamic dispatch)
By default, Fory auto-detects polymorphism via `std::is_polymorphic<T>`. Use this tag to override:
```cpp
// Base class with virtual methods (detected as polymorphic by default)
struct Animal {
virtual ~Animal() = default;
virtual std::string speak() const = 0;
};
struct Zoo {
// Auto: type info written because Animal has virtual methods
fory::field<std::shared_ptr<Animal>, 0, fory::nullable> animal;
// Force non-dynamic: skip type info even though Animal has virtual methods
// Use when you know the runtime type will always be exactly as declared
fory::field<std::shared_ptr<Animal>, 1, fory::nullable, fory::dynamic<false>> fixed_animal;
};
FORY_STRUCT(Zoo, animal, fixed_animal);
```
**Valid for:** `std::shared_ptr<T>`, `std::unique_ptr<T>`
### Combining Tags
Multiple tags can be combined for shared pointers:
```cpp
// Nullable + ref tracking
fory::field<std::shared_ptr<Node>, 0, fory::nullable, fory::ref> link;
```
## Type Rules
| Type | Allowed Options | Nullability |
| -------------------- | ------------------------------- | ---------------------------------- |
| Primitives, strings | None | Use `std::optional<T>` if nullable |
| `std::optional<T>` | None | Inherently nullable |
| `std::shared_ptr<T>` | `nullable`, `ref`, `dynamic<V>` | Non-null by default |
| `std::unique_ptr<T>` | `nullable`, `dynamic<V>` | Non-null by default |
## Complete Example
```cpp
#include "fory/serialization/fory.h"
using namespace fory::serialization;
// Define a struct with various field configurations
struct Document {
// Required fields (non-nullable)
fory::field<std::string, 0> title;
fory::field<int32_t, 1> version;
// Optional primitive using std::optional
fory::field<std::optional<std::string>, 2> description;
// Nullable pointer
fory::field<std::unique_ptr<std::string>, 3, fory::nullable> metadata;
// Reference-tracked shared pointer
fory::field<std::shared_ptr<Document>, 4, fory::ref> parent;
// Nullable + reference-tracked
fory::field<std::shared_ptr<Document>, 5, fory::nullable, fory::ref> related;
};
FORY_STRUCT(Document, title, version, description, metadata, parent, related);
int main() {
auto fory = Fory::builder().xlang(true).build();
fory.register_struct<Document>(100);
Document doc;
doc.title = "My Document";
doc.version = 1;
doc.description = "A sample document";
doc.metadata = nullptr; // Allowed because nullable
doc.parent = std::make_shared<Document>();
doc.parent->title = "Parent Doc";
doc.related = nullptr; // Allowed because nullable
auto bytes = fory.serialize(doc).value();
auto decoded = fory.deserialize<Document>(bytes).value();
}
```
## Compile-Time Validation
Invalid configurations are caught at compile time:
```cpp
// Error: nullable and not_null are mutually exclusive
fory::field<std::shared_ptr<int>, 0, fory::nullable, fory::not_null> bad1;
// Error: nullable only valid for smart pointers
fory::field<int32_t, 0, fory::nullable> bad2;
// Error: ref only valid for shared_ptr
fory::field<std::unique_ptr<int>, 0, fory::ref> bad3;
// Error: options not allowed for std::optional (inherently nullable)
fory::field<std::optional<int>, 0, fory::nullable> bad4;
```
## Backwards Compatibility
Existing structs without `fory::field<>` wrappers continue to work:
```cpp
// Old style - still works
struct LegacyPerson {
std::string name;
int32_t age;
};
FORY_STRUCT(LegacyPerson, name, age);
// New style with field metadata
struct ModernPerson {
fory::field<std::string, 0> name;
fory::field<int32_t, 1> age;
};
FORY_STRUCT(ModernPerson, name, age);
```
## FORY_FIELD_TAGS Macro
The `FORY_FIELD_TAGS` macro provides a non-invasive way to add field metadata without modifying struct definitions. This is useful for:
- **Third-party types**: Add metadata to types you don't own
- **Clean structs**: Keep struct definitions as pure C++
- **Isolated dependencies**: Confine Fory headers to serialization config files
### Usage
```cpp
// user_types.h - NO fory headers needed!
struct Document {
std::string title;
int32_t version;
std::optional<std::string> description;
std::shared_ptr<User> author;
std::shared_ptr<User> reviewer;
std::shared_ptr<Document> parent;
std::unique_ptr<Data> data;
};
// serialization_config.cpp - fory config isolated here
#include "fory/serialization/fory.h"
#include "user_types.h"
FORY_STRUCT(Document, title, version, description, author, reviewer, parent, data)
FORY_FIELD_TAGS(Document,
(title, 0), // string: non-nullable
(version, 1), // int: non-nullable
(description, 2), // optional: inherently nullable
(author, 3), // shared_ptr: non-nullable (default)
(reviewer, 4, nullable), // shared_ptr: nullable
(parent, 5, ref), // shared_ptr: non-nullable, with ref tracking
(data, 6, nullable) // unique_ptr: nullable
)
```
### FORY_FIELD_TAGS Options
| Field Type | Valid Combinations |
| -------------------- | ---------------------------------------------------------------------------------------- |
| Primitives, strings | `(field, id)` only |
| `std::optional<T>` | `(field, id)` only |
| `std::shared_ptr<T>` | `(field, id)`, `(field, id, nullable)`, `(field, id, ref)`, `(field, id, nullable, ref)` |
| `std::unique_ptr<T>` | `(field, id)`, `(field, id, nullable)` |
### API Comparison
| Aspect | `fory::field<>` Wrapper | `FORY_FIELD_TAGS` Macro |
| ----------------------- | ------------------------ | ----------------------- |
| **Struct definition** | Modified (wrapped types) | Unchanged (pure C++) |
| **IDE support** | Template noise | Excellent (clean types) |
| **Third-party classes** | Not supported | Supported |
| **Header dependencies** | Required everywhere | Isolated to config |
| **Migration effort** | High (change all fields) | Low (add one macro) |
## FORY_FIELD_CONFIG Macro
The `FORY_FIELD_CONFIG` macro is the most powerful and flexible way to configure field-level serialization. It provides:
- **Builder pattern API**: Fluent, chainable configuration with `F(id).option1().option2()`
- **Encoding control**: Specify how unsigned integers are encoded (varint, fixed, tagged)
- **Compile-time verification**: Field names are verified against member pointers
- **Cross-language compatibility**: Configure encoding to match other languages (Java, Rust, etc.)
### Basic Syntax
```cpp
FORY_FIELD_CONFIG(StructType,
(field1, fory::F(0)), // Simple: just ID
(field2, fory::F(1).nullable()), // With nullable
(field3, fory::F(2).varint()), // With encoding
(field4, fory::F(3).nullable().ref()), // Multiple options
(field5, 4) // Backward compatible: integer ID
);
```
### The F() Builder
The `fory::F(id)` factory creates a `FieldMeta` object that supports method chaining:
```cpp
fory::F(0) // Create with field ID 0
.nullable() // Mark as nullable
.ref() // Enable reference tracking
.varint() // Use variable-length encoding
.fixed() // Use fixed-size encoding
.tagged() // Use tagged encoding
.dynamic(false) // Skip type info (no dynamic dispatch)
.dynamic(true) // Force type info (enable dynamic dispatch)
.compress(false) // Disable compression
```
**Tip:** To use `F()` without the `fory::` prefix, add a using declaration:
```cpp
using fory::F;
FORY_FIELD_CONFIG(MyStruct,
(field1, F(0).varint()), // No prefix needed
(field2, F(1).nullable())
);
```
### Encoding Options for Unsigned Integers
For `uint32_t` and `uint64_t` fields, you can specify the wire encoding:
| Method | Type ID | Description | Use Case |
| ----------- | ------------- | ---------------------------------------------- | ------------------------------------- |
| `.varint()` | VAR_UINT32/64 | Variable-length encoding (1-5 or 1-10 bytes) | Values typically small |
| `.fixed()` | UINT32/64 | Fixed-size encoding (always 4 or 8 bytes) | Values uniformly distributed |
| `.tagged()` | TAGGED_UINT64 | Tagged hybrid encoding with size hint (uint64) | Mixed small and large values (uint64) |
**Note:** `uint8_t` and `uint16_t` always use fixed encoding (UINT8, UINT16).
### Complete Example
```cpp
#include "fory/serialization/fory.h"
using namespace fory::serialization;
// Define struct with unsigned integer fields
struct MetricsData {
// Counters - often small values, use varint for space efficiency
uint32_t requestCount;
uint64_t bytesSent;
// IDs - uniformly distributed, use fixed for consistent performance
uint32_t userId;
uint64_t sessionId;
// Timestamps - use tagged encoding for mixed value ranges
uint64_t createdAt;
// Nullable fields
std::optional<uint32_t> errorCount;
std::optional<uint64_t> lastAccessTime;
};
FORY_STRUCT(MetricsData, requestCount, bytesSent, userId, sessionId,
createdAt, errorCount, lastAccessTime);
// Configure field encoding
FORY_FIELD_CONFIG(MetricsData,
// Small counters - varint saves space
(requestCount, fory::F(0).varint()),
(bytesSent, fory::F(1).varint()),
// IDs - fixed for consistent performance
(userId, fory::F(2).fixed()),
(sessionId, fory::F(3).fixed()),
// Timestamp - tagged encoding
(createdAt, fory::F(4).tagged()),
// Nullable fields
(errorCount, fory::F(5).nullable().varint()),
(lastAccessTime, fory::F(6).nullable().tagged())
);
int main() {
auto fory = Fory::builder().xlang(true).build();
fory.register_struct<MetricsData>(100);
MetricsData data{
.requestCount = 42,
.bytesSent = 1024,
.userId = 12345678,
.sessionId = 9876543210,
.createdAt = 1704067200000000000ULL, // 2024-01-01 in nanoseconds
.errorCount = 3,
.lastAccessTime = std::nullopt
};
auto bytes = fory.serialize(data).value();
auto decoded = fory.deserialize<MetricsData>(bytes).value();
}
```
### Cross-Language Compatibility
When serializing data to be read by other languages, use `FORY_FIELD_CONFIG` to match their encoding expectations:
**Java Compatibility:**
```cpp
// Java uses these type IDs for unsigned integers:
// - Byte (u8): UINT8 (fixed)
// - Short (u16): UINT16 (fixed)
// - Integer (u32): VAR_UINT32 (varint) or UINT32 (fixed)
// - Long (u64): VAR_UINT64 (varint), UINT64 (fixed), or TAGGED_UINT64
struct JavaCompatible {
uint8_t byteField; // Maps to Java Byte
uint16_t shortField; // Maps to Java Short
uint32_t intVarField; // Maps to Java Integer with varint
uint32_t intFixedField; // Maps to Java Integer with fixed
uint64_t longVarField; // Maps to Java Long with varint
uint64_t longTagged; // Maps to Java Long with tagged
};
FORY_STRUCT(JavaCompatible, byteField, shortField, intVarField,
intFixedField, longVarField, longTagged);
FORY_FIELD_CONFIG(JavaCompatible,
(byteField, fory::F(0)), // UINT8 (auto)
(shortField, fory::F(1)), // UINT16 (auto)
(intVarField, fory::F(2).varint()), // VAR_UINT32
(intFixedField, fory::F(3).fixed()), // UINT32
(longVarField, fory::F(4).varint()), // VAR_UINT64
(longTagged, fory::F(5).tagged()) // TAGGED_UINT64
);
```
### Schema Evolution with FORY_FIELD_CONFIG
In compatible mode, fields can have different nullability between sender and receiver:
```cpp
// Version 1: All fields non-nullable
struct DataV1 {
uint32_t id;
uint64_t timestamp;
};
FORY_STRUCT(DataV1, id, timestamp);
FORY_FIELD_CONFIG(DataV1,
(id, fory::F(0).varint()),
(timestamp, fory::F(1).tagged())
);
// Version 2: Added nullable fields
struct DataV2 {
uint32_t id;
uint64_t timestamp;
std::optional<uint32_t> version; // New nullable field
};
FORY_STRUCT(DataV2, id, timestamp, version);
FORY_FIELD_CONFIG(DataV2,
(id, fory::F(0).varint()),
(timestamp, fory::F(1).tagged()),
(version, fory::F(2).nullable().varint()) // New field with nullable
);
```
### FORY_FIELD_CONFIG Options Reference
| Method | Description | Valid For |
| ----------------- | ------------------------------------------------ | -------------------------- |
| `.nullable()` | Mark field as nullable | Smart pointers, primitives |
| `.ref()` | Enable reference tracking | `std::shared_ptr` only |
| `.dynamic(true)` | Force type info to be written (dynamic dispatch) | Smart pointers |
| `.dynamic(false)` | Skip type info (use declared type directly) | Smart pointers |
| `.varint()` | Use variable-length encoding | `uint32_t`, `uint64_t` |
| `.fixed()` | Use fixed-size encoding | `uint32_t`, `uint64_t` |
| `.tagged()` | Use tagged hybrid encoding | `uint64_t` only |
| `.compress(v)` | Enable/disable field compression | All types |
### Comparing Field Configuration Macros
| Feature | `fory::field<>` | `FORY_FIELD_TAGS` | `FORY_FIELD_CONFIG` |
| ----------------------- | --------------------- | ----------------- | ------------------------- |
| **Struct modification** | Required (wrap types) | None | None |
| **Encoding control** | No | No | Yes (varint/fixed/tagged) |
| **Builder pattern** | No | No | Yes |
| **Compile-time verify** | Yes | Limited | Yes (member pointers) |
| **Cross-lang compat** | Limited | Limited | Full |
| **Recommended for** | Simple structs | Third-party types | Complex/xlang structs |
## Default Values
- **Nullable**: Only `std::optional<T>` is nullable by default; all other types (including `std::shared_ptr`) are non-nullable
- **Ref tracking**: Disabled by default for all types (including `std::shared_ptr`)
You **need to configure fields** when:
- A field can be null (use `std::optional<T>` or mark with `nullable()`)
- A field needs reference tracking for shared/circular objects (use `ref()`)
- Integer types need specific encoding for cross-language compatibility
- You want to reduce metadata size (use field IDs)
```cpp
// Xlang mode: explicit configuration required
struct User {
std::string name; // Non-nullable by default
std::optional<std::string> email; // Nullable (std::optional)
std::shared_ptr<User> friend_ptr; // Ref tracking by default
};
FORY_STRUCT(User, name, email, friend_ptr);
FORY_FIELD_CONFIG(User,
(name, fory::F(0)),
(email, fory::F(1)), // nullable implicit for optional
(friend_ptr, fory::F(2).nullable().ref()) // explicit nullable + ref
);
```
### Default Values Summary
| Type | Default Nullable | Default Ref Tracking |
| -------------------- | ---------------- | -------------------- |
| Primitives, `string` | `false` | `false` |
| `std::optional<T>` | `true` | `false` |
| `std::shared_ptr<T>` | `false` | `true` |
| `std::unique_ptr<T>` | `false` | `false` |
## Related Topics
- [Type Registration](type-registration.md) - Registering types with FORY_STRUCT
- [Schema Evolution](schema-evolution.md) - Using tag IDs for schema evolution
- [Configuration](configuration.md) - Enabling reference tracking globally
- [Cross-Language](cross-language.md) - Interoperability with Java, Rust, Python