Add an implementation of the Raft consensus protocol (#223) * Bumped zlib version fixed a race condition in deploy_local.sh * Added Raft files, currently copies of PoE files per Dakai's guide. * set PoE script to use pbft's kvtools * Replace view change flag with leader election flag * WIP add initial happy path raft implementation * WIP some bug fixes * WIP log statements to try and figure things out * WIP fix using wrong value for leader id * WIP more print statements * Fixed errors in build files to support Intellisense * WIP temp change to broadcast * WIP add more prints, leader commits and executes * WIP switch back to SendMessage * WIP fixed committing condition * Added Heartbeat and Elections triggered by timeout * Update variable names to match the Raft paper * in progress requestVote and requestVoteResponse handlers * finished adding leader election logic, it seems buggy though. * blocked input from client, set appendEntries handler to always be happy raised timeout window and heartbeat interval * testing * testing leader election * staging for merge into raft branch * Added logic to redirect client proxy to current leader * Fix AppendEntries failure path to back off nextIndex and retry * redirect to leader working even on later terms * seems to kind of work. worried about duplicate entries? * modified some logging, changed config * moved some variables around in header, changed receivePropose to ReceiveAppendEntries * Swapped to requests instead of AppendEntries outside RPC logic * implemented LogEntry struct instead of using AppendEntries for everything * Create and send individual AppendEntries to each follower from ReceiveTransaction * Setting nextIndex to lastlogindex() - 1 in ReceiveAEResponse fail case * Modified CreateAndSendAppendEntries to CreateAndSendAppendEntryMsg and used it in ReceiveAppendEntriesResponse * Fix issues from the previous commit * WIP catch up transactions to leader's most current log entry * staging for benchmarks * Added support for multiple entries in AppendEntries RPC Heartbeat is skipped if leader recently broadcasted AppendEntries Added logic for followers to truncate their log when leader sends conflicting entries * added followerId field to AeFields struct to simplify code changed variable names for semanatic value removed one line conditionals for readability * revamped follower truncate/append logic * added some comments to Raft::ReceiveAppendEntries method * update inflight limit code * modified config generation script to accomodate changes to config reading in upstream code * Add a function for use in later commit * Add tests related to timeouts and heartbeats for Raft * Add initial Raft tests for sending and receiving AppendEntries * Update tests, restructure raft tests to support multiple raft test files * Add tests for ReceiveAppendEntriesResponse() * Add tests related to voting * Run clang format and other style changes * Undo some changes unrelated to Raft * Change log_ to vector<LogEntry>. Add helper functions for future use * Change LogEntry to store Entry directly * WIP Add RaftRecovery class * WIP Extract common Recovery code to BaseRecovery * WIP Update RaftRecovery and tests, tentatively working * Run clang format * Change the templating of Recovery * Reorganize file structure of recovery files * Update things missed in previous commit * Fix RaftRecovery issue, add test for it, adjust dependencies * Remove access modifier change for test compilation, change whitespace * Address some PR comments, fix broken tests Tests now work again after adding Recovery. Addressed minor test concerns. Add Log Truncation support to Raft Recovery. This includes a new WALRecord and TruncationRecord protobuf message. Move LastLogIndex and log_ resizing into AddToLog and TruncateLog. Add PrintDebugStateLocked() so that PrintDebugState() can be called under the lock. Change Recovery to not ignore all recovery data if a portion of the data is wrong. * Improve reliability of RaftRecovery * Update changes for tests, change how log is accessed For checkpointing, we need to separate the logical log index and size from the actual indexing and size of the vector. * Add Checkpointing, new tests, miscellaneous changes Added some handling to deal with snapshotting in Raft. Checkpoints will be taken, the last snapshot index and term need to be stored, and index arithmetic needs to be accounted for. Note that this largely is not yet in use, as the hook to actual truncate a prefix in Consensus is currently commented out. Once there is handling to send and receive Snapshots, this will be used. Added a callback hook in recovery to let the consensus protocol know when a checkpoint has finished and up to what seq it covers. Moved some Raft test header functionality into a general Raft test utility to not be attached to that specific text fixture. Added some new tests for existing Raft features. Updated lastApplied_ in Raft to be lastCommitted (meaning we have queued it to be committed). lastApplied is now tracked in Raft's Consensus class. Fixed a bug with min_seq_ tracking in Recovery. Fixed potential issues with edge cases in ReadLogsFromFiles. * Undo change to recovery_ckpt_time_s_ --------- Co-authored-by: Jim Brower <jimrabrower@gmail.com> Co-authored-by: nachiket <nachocheeeze922@gmail.com> Co-authored-by: yhuan331 <yhuan331@ucsc.edu> Co-authored-by: hammerface <hammerface@users.noreply.github.com>
ResilientDB is a High Throughput Yielding Permissioned Blockchain Fabric founded by ExpoLab at UC Davis in 2018. ResilientDB advocates a system-centric design by adopting a multi-threaded architecture that encompasses deep pipelines. Further, ResilientDB separates the ordering of client transactions from their execution, which allows it to process messages out-of-order.
Download address for run-directly software package: https://downloads.apache.org/incubator/resilientdb/
ecosystem/ directory, including wallets (ResVault), SDKs and ORMs (Python, Rust, ResDB‑ORM), APIs (GraphQL, REST), and monitoring & deployment tooling (ResLens, Ansible/Docker, Orbit, etc.), making it easy to build, observe, and operate applications on top of ResilientDB.The latest ResilientDB documentation is available on Beacon, our unified documentation portal.
For a deep, chapter‑wise explanation of how ResilientDB works, see the ResilientDB overview and chapters.
Historical and supplementary guides are also available on our blog repository.
Beyond the core NexRes engine, ResilientDB ships with a rich ecosystem of tools that make it easier to build applications and data/AI pipelines on top of the ledger:
Together, these components provide an end-to-end path from user-facing applications (web, services, or AI agents) through wallets and APIs down to the ResilientDB core.
Ubuntu 20+
incubator-resilientdb/ ├── api/ # API layer and interfaces ├── benchmark/ # Performance benchmarking tools │ └── protocols/ # Protocol-specific benchmarks │ ├── pbft/ # PBFT protocol benchmarks │ └── poe/ # PoE protocol benchmarks ├── chain/ # Blockchain chain management │ ├── state/ # Chain state management │ └── storage/ # Storage layer (LevelDB, etc.) ├── common/ # Common utilities and libraries │ ├── crypto/ # Cryptographic functions │ ├── lru/ # LRU cache implementation │ ├── proto/ # Protocol buffer definitions │ ├── test/ # Testing utilities │ └── utils/ # General utilities ├── ecosystem/ # Ecosystem components (git subtrees) │ ├── ai-tools/ # AI, docs, and MCP tools │ │ ├── beacon/ # Beacon docs site │ │ ├── nexus/ # Nexus research / LLM tools │ │ └── mcp/ # MCP integrations (GraphQL, ResInsight) │ ├── cache/ # Caching libraries │ │ ├── resilient-node-cache/ # Node.js cache │ │ └── resilient-python-cache/ # Python cache │ ├── deployment/ # Deployment tooling │ │ ├── ansible/ # Ansible playbooks │ │ └── orbit/ # Orbit desktop deployer │ ├── graphql/ # GraphQL + HTTP services │ ├── monitoring/ # Monitoring stack │ │ ├── reslens/ # ResLens UI │ │ └── reslens-middleware/ # ResLens backend │ ├── sdk/ # SDKs and ORMs │ │ ├── resdb-orm/ # Python ORM │ │ ├── resvault-sdk/ # ResVault SDK │ │ └── rust-sdk/ # Rust SDK │ ├── smart-contract/ # Smart contract stack │ │ ├── rescontract/ # Contracts and examples │ │ ├── resilient-contract-kit/ # Contract dev toolkit │ │ └── smart-contract-graphql/ # Contract GraphQL service │ ├── third_party/ # External tools (e.g., PocketFlow) │ └── tools/ # Dev and app tools │ ├── create-resilient-app/ # App scaffolding CLI │ ├── drawing-lib/ # Shared drawing widget │ ├── reshare-lib/ # ResShare SDK │ └── resvault/ # ResVault wallet extension ├── executor/ # Transaction execution engine │ ├── common/ # Common execution utilities │ ├── contract/ # Smart contract execution │ ├── kv/ # Key-value execution │ └── utxo/ # UTXO execution ├── interface/ # Client interfaces and APIs │ ├── common/ # Common interface utilities │ ├── contract/ # Smart contract interface │ ├── kv/ # Key-value interface │ ├── rdbc/ # ResilientDB Database Connectivity │ └── utxo/ # UTXO interface ├── monitoring/ # Core monitoring components ├── platform/ # Core platform components │ ├── common/ # Common platform utilities │ ├── config/ # Configuration management │ ├── consensus/ # Consensus protocols │ │ ├── checkpoint/ # Checkpoint management │ │ ├── execution/ # Transaction execution │ │ ├── ordering/ # Transaction ordering │ │ └── recovery/ # Recovery mechanisms │ ├── networkstrate/ # Network strategy layer │ ├── proto/ # Protocol definitions │ ├── rdbc/ # RDBC implementation │ └── statistic/ # Statistics and metrics ├── proto/ # Protocol buffer definitions │ ├── contract/ # Smart contract protos │ ├── kv/ # Key-value protos │ └── utxo/ # UTXO protos ├── scripts/ # Deployment and utility scripts │ └── deploy/ # Deployment scripts ├── service/ # Service implementations │ ├── contract/ # Smart contract service │ ├── kv/ # Key-value service │ ├── tools/ # Service tools │ ├── utils/ # Service utilities │ └── utxo/ # UTXO service ├── third_party/ # Third-party dependencies └── tools/ # Development and build tools
Note: The ecosystem/ directory contains git subtrees for ecosystem components. You can clone the repository without ecosystem components for a smaller, faster download. See ecosystem/README.md for details.
Next, we show how to quickly build ResilientDB and deploy 4 replicas and 1 client proxy on your local machine. The proxy acts as an interface for all the clients. It batches client requests and forwards these batches to the replica designated as the leader. The 4 replicas participate in the PBFT consensus to order and execute these batches. Post execution, they return the response to the leader.
Install dependencies:
./INSTALL.sh
For non-root users, see INSTALL/README.md
Generate the certificates and the public-private keys.
./service/tools/kv/server_tools/generate_keys_and_certs.sh
Run ResilientDB (Providing a Key-Value Service):
./service/tools/kv/server_tools/start_kv_service.sh
Build Interactive Tools:
bazel build service/tools/kv/api_tools/kv_service_tools
If you cannot build the project successfully, try to reduce the bazel jobs here.
ResilientDB supports two types of functions: version-based and non-version-based. Version-based functions will leverage versions to protect each update, versions must be obtained before updating a key.
Note: Version-based functions are not compatible with non-version-based functions. Do not use both in your applications.
We show the functions below and show how to use kv_service_tools to test the function.
Obtain the value of key with a specific version v.
kv_service_tools --config config_file --cmd get_with_version --key key --version v
| parameters | descriptions |
|---|---|
| config | the path of the client config which points to the db entrance |
| cmd | get_with_version |
| key | the key you want to obtain |
| version | the version you want to obtain. (If the v is 0, it will return the latest version |
Example:
bazel-bin/service/tools/kv/api_tools/kv_service_tools --config service/tools/config/interface/service.config --cmd get_with_version --key key1 --version 0
Results:
get key = key1, value = value: “v2” version: 2
Set value to the key key based on version v.
kv_service_tools --config config_file --cmd set_with_version --key key --version v --value value
| parameters | descriptions |
|---|---|
| config | the path of the client config which points to the db entrance |
| cmd | set_with_version |
| key | the key you want to set |
| version | the version you have obtained. (If the version has been changed during the update, the transaction will be ignored) |
| value | the new value |
Example:
bazel-bin/service/tools/kv/api_tools/kv_service_tools --config service/tools/config/interface/service.config --cmd set_with_version --key key1 --version 0 --value v1
Results:
set key = key1, value = v3, version = 2 done, ret = 0
current value = value: “v3” version: 3
Obtain the update history of key key within the versions [v1, v2].
kv_service_tools --config config_file --cmd get_history --key key --min_version v1 --max_version v2
| parameters | descriptions |
|---|---|
| config | the path of the client config which points to the db entrance |
| cmd | get_history |
| key | the key you want to obtain |
| min_version | the minimum version you want to obtain |
| max_version | the maximum version you want to obtain |
Example:
bazel-bin/service/tools/kv/api_tools/kv_service_tools --config service/tools/config/interface/service.config --cmd get_history --key key1 --min_version 1 --max_version 2
Results:
get history key = key1, min version = 1, max version = 2
value =
item {
key: “key1”
value_info {
value: “v1”
version: 2
}
}
item {
key: “key1”
value_info {
value: “v0”
version: 1
}
}
Obtain the recent top_number history of the key key.
kv_service_tools --config config_path --cmd get_top --key key --top top_number
| parameters | descriptions |
|---|---|
| config | the path of the client config which points to the db entrance |
| cmd | get_top |
| key | the key you want to obtain |
| top | the number of the recent updates |
Example:
bazel-bin/service/tools/kv/api_tools/kv_service_tools --config service/tools/config/interface/service.config --cmd get_top --key key1 --top 1
Results:
key = key1, top 1
value =
item {
key: “key1”
value_info {
value: “v2”
version: 3
}
}
Obtain the values of the keys in the ranges [key1, key2]. Do not use this function in your practice code
kv_service_tools --config config_file --cmd get_key_range_with_version --min_key key1 --max_key key2
| parameters | descriptions |
|---|---|
| config | the path of the client config which points to the db entrance |
| cmd | get_key_range_with_version |
| min_key | the minimum key |
| max_key | the maximum key |
Example:
bazel-bin/service/tools/kv/api_tools/kv_service_tools --config service/tools/config/interface/service.config --cmd get_key_range_with_version --min_key key1 --max_key key3
Results:
min key = key1 max key = key2
getrange value =
item {
key: “key1”
value_info {
value: “v0”
version: 1
}
}
item {
key: “key2”
value_info {
value: “v1”
version: 1
}
}
Set value to the key key.
kv_service_tools --config config_file --cmd set --key key --value value
| parameters | descriptions |
|---|---|
| config | the path of the client config which points to the db entrance |
| cmd | set |
| key | the key you want to set |
| value | the new value |
Example:
bazel-bin/service/tools/kv/api_tools/kv_service_tools --config service/tools/config/interface/service.config --cmd set --key key1 --value value1
Results:
set key = key1, value = v1, done, ret = 0
Obtain the value of key.
kv_service_tools --config config_file --cmd get --key key
| parameters | descriptions |
|---|---|
| config | the path of the client config which points to the db entrance |
| cmd | get |
| key | the key you want to obtain |
Example:
bazel-bin/service/tools/kv/api_tools/kv_service_tools --config service/tools/config/interface/service.config --cmd get --key key1
Results:
get key = key1, value = “v2”
Obtain the values of the keys in the ranges [key1, key2]. Do not use this function in your practice code
kv_service_tools --config config_path --cmd get_key_range --min_key key1 --max_key key2
| parameters | descriptions |
|---|---|
| config | the path of the client config which points to the db entrance |
| cmd | get_key_range |
| min_key | the minimum key |
| max_key | the maximum key |
Example:
bazel-bin/service/tools/kv/api_tools/kv_service_tools --config service/tools/config/interface/service.config --cmd get_key_range --min_key key1 --max_key key3
Results:
getrange min key = key1, max key = key3
value = [v3,v2,v1]
We also provide access to a deployment script that allows deployment on distinct machines.
Install Docker
Before getting started, make sure you have Docker installed on your system. If you don't have Docker already, you can download and install it from the official Docker website.
Pull the Latest ResilientDB Image
Choose the appropriate ResilientDB image for your machine's architecture:
For amd architecture, run:
docker pull expolab/resdb:amd64
For Apple Silicon (M1/M2) architecture, run:
docker pull expolab/resdb:arm64
Run a Container with the Pulled Image
Launch a Docker container using the ResilientDB image you just pulled:
For amd architecture, run:
docker run -d --name myserver expolab/resdb:amd64
For Apple Silicon (M1/M2) architecture, run:
docker run -d --name myserver expolab/resdb:arm64
Test with Set and Get Commands Exec into the running server:
docker exec -it myserver bash
NOTE: If you encounter a Connection Refused error
Run the following command within the container:
./service/tools/kv/server_tools/start_kv_service.sh
Verify the functionality of the service by performing set and get operations provided above functions.
When starting the service locally, current services are running on 10000 port-base with 5 services where the server config is located here
If you want to change the setting, you need to generate the certificates.
Go the the workspace where the resilientdb repo is localted.
Change the setting parameters here and run the script:
./service/tools/kv/server_tools/generate_config.sh
Then re-run the start script:
./service/tools/kv/server_tools/start_kv_service.sh
ResilientDB includes a built‑in smart contract service that lets you deploy and execute Solidity contracts on top of NexRes. There are two primary ways to work with smart contracts:
Low‑level flow via contract_service_tools
This flow uses the contract_service_tools client and JSON payloads:
contract_service_tools with a JSON file containing {"command": "create_account"} to obtain an on‑chain owner address.token.sol).solc, for example:solc --evm-version homestead --combined-json bin,hashes --pretty-json --optimize token.sol > token.jsoncommand: "deploy"contract_path: path to the compiled JSON (e.g., service/tools/kv/api_tools/example_contract/token.json)contract_name: contract identifier (e.g., "token.sol:Token")init_params: constructor parameters (if any)owner_address: the owner account addresscontract_service_tools with a JSON payload that includes:command: "execute"contract_address: deployed contract addresscaller_address: account invoking the functionfunc_name: function signature (e.g., "transfer(address,uint256)")params: encoded parametersHigher‑level tools: ResContract CLI and ResVault
Instead of crafting JSON by hand, you can use tools from the ecosystem/ directory that wrap this functionality:
ecosystem/smart-contract/rescontract/):ecosystem/tools/resvault/):These tools provide full functionality up to and including deployment and interaction with smart contracts. See their individual READMEs for detailed usage and examples.
In addition, you can use the Key‑Value interfaces to query or update associated balances via get_balance and set_balance commands when needed.
For a full, step‑by‑step tutorial (including example JSON files, commands, and responses), see Getting Started with Smart Contract on ResilientDB.