apache-iggy

Apache Iggy is the persistent message streaming platform written in Rust, supporting QUIC, TCP and HTTP transport protocols, capable of processing millions of messages per second.

Installation

Basic Installation

# Using uv in an existing project
uv add apache-iggy

# Using pip
python3 -m venv .venv
source .venv/bin/activate
pip install apache-iggy

Prerequisites

Every installation below compiles the Rust extension, so you'll need:

Python 3.10+
Rust toolchain: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
uv: curl -LsSf https://astral.sh/uv/install.sh | sh
All checks tooling from CONTRIBUTING.md.
Docker

Local Development

IMPORTANT: All commands are supposed to be ran from foreign/python unless it‘s specified to run in repository’s root folder.

Build a project for development

With uv:

# Create a venv
uv venv

# Sync the environment without updating it
uv sync --frozen --all-extras --no-install-project

# Build the project -- this builds the rust extension into the venv (debug profile) - re-run after any rust change
uv run --no-sync maturin develop

With pip:

# Create a venv
python3 -m venv .venv

# Activate the venv
source .venv/bin/activate

# Install the dependencies
pip install -e ".[all]"

# Build the project -- this builds the rust extension into the venv (debug profile) - re-run after any rust change
maturin develop

Run the server to be able to run the tests (this blocks the terminal - run steps 3-5 in a separate one). --fresh deletes local_data/ on every run - drop it if you have existing data you want to keep.
```
# run from the repository's root directory
cargo run --bin iggy-server -- --with-default-root-credentials --fresh
```

Run the tests

uv:

uv run --no-sync pytest tests/ -v

pip:

pytest tests/ -v # make sure iggy-server is running and the venv is activated

To update the stubs, after changing the pyo3 API surface, use
```
# run from foreign/python
cargo run --bin stub_gen
```

Before committing, test the pre-commit and pre-push hooks. prek only inspects staged content, so stage your work first:

git add -A
prek run # runs pre-commit hooks
prek run --hook-stage pre-push
# if a hook modifies files, re-run `git add -A` and `prek run`.

These are some of the essential commands prek is running, so it's recommended to run them manually before running prek / committing / pushing. This list is not exhaustive and other hook failures are possible.

uv run --no-sync ruff format .

uv run --no-sync ruff check --fix .

cargo fmt --manifest-path Cargo.toml

cargo clippy --manifest-path Cargo.toml --all-targets --all-features -- -D warnings

# run from the repository's root directory
./scripts/ci/markdownlint.sh --fix foreign/python/README.md # read the diff after applying this, sometimes it gives unwanted results, e.g. messing up enumerations

Client Configuration

IggyClient takes a server address, a TcpConfig, a QuicConfig, an HttpConfig, or a WebSocketConfig:

import asyncio
from datetime import timedelta

from apache_iggy import AutoLogin, IggyClient, TcpConfig, TcpReconnectionConfig


async def main():
    client = IggyClient(
        TcpConfig(
            server_address="127.0.0.1:8090",
            auto_login=AutoLogin.username_password("iggy", "iggy"),
            reconnection=TcpReconnectionConfig(
                enabled=True,
                max_retries=10,
                interval=timedelta(seconds=2),
                reestablish_after=timedelta(seconds=30),
            ),
            heartbeat_interval=timedelta(seconds=5),
            # tls_enabled=True,
            # tls_domain="localhost",
            # tls_ca_file="../../core/certs/iggy_ca_cert.pem",
            # tls_validate_certificate=True,
            # nodelay=True,
        )
    )
    await client.connect()


asyncio.run(main())

IggyClient(...) also accepts a QuicConfig for the QUIC transport, an HttpConfig for the HTTP transport, and a WebSocketConfig for the WebSocket transport. examples/python/getting-started/producer.py shows each swap in context.

HttpConfig differs from TCP in two ways. There is no reconnection policy and no AutoLogin: connect() does not dial over HTTP, but it does start the heartbeat that heartbeat_interval configures, so call it and then login_user(...). And HTTP is single-consumer only: the consumer_group(...) path always fails with Feature is unavailable, at the join by default and at the returned consumer's first poll if you disable auto_join_consumer_group, so disabling it is not a workaround. A direct poll_messages(consumer=Consumer.Group(...)) fails the same way unless you pass an explicit partition_id, and with one it degrades silently instead: the consumer kind is not carried on the HTTP wire, so the group is served as an ordinary consumer named after it, with no membership or partition assignment behind it. Use Consumer.Single(...) with poll_messages(...). Delivery is also at-least-once: the default retries=3 replays the full request body, so a send whose response was lost is applied twice, and only retries=0 opts out.

import asyncio

from apache_iggy import HttpConfig, IggyClient


async def main():
    client = IggyClient(HttpConfig(api_url="http://127.0.0.1:3000"))
    await client.connect()
    await client.login_user("iggy", "iggy")


asyncio.run(main())

Examples

Refer to the examples/python/ directory for usage examples.

Contributing

See CONTRIBUTING.md for contribution guidelines.

License

Licensed under the Apache License 2.0. See LICENSE for details.